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FOR^ORD 



Timely and accurate evaluations of social programs are issential 
to sound planning and public policy deciiions at loc^, state, and 
national levels. The need is clearly evident m the field of 
employment-related programs, . which involve the expendimrc of 
biilfons of dollars and affect the lives of millions of individuals. 

This primer presents an approach and methodology for the 
systematic measuremem of the impact of employmcnt-rdated 
social progiams. The primary emphasis is on basic techmques of 
evaluation, with references to numerous theoretic^ and. con- 
ceptual issues. This guide should add sigr.ificantly to the hteraturc 
on program evaluation and is intended to assist those, who conduct 
impact evaluations as well as program planners and admimstrators 
who must make decisions based on such evaluations. 

Facts and ubservationi as presented in this monograph are the 
sole responsibility of the author. His viewpoints do neces^nbj 
represe^ positions of the W. E. Upjohn Institute for Employment 
Research. 



E. Earl Wright 
Director 



Kalamazoo, Michigan 
April im 
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PRErACK 



This book is 1^ reviled version of an earlier woric, Memwing the 
Impact af Maftpomr Programs: A Primer, written by tiie present 
author and WlUiam R. Taoh and published by the Institute of 
Labor and IndustrliiL Relations, The University of Michigan^ 
Wayne State University as Policy Papers in Human Resources and 
Uidusirlal Relations No. 17. The present volume has been changed 
ronsiderflbly from the earlier primer, rhesc changes arc based on 
the development of the f ield in the »cn years fdncc the first worIc 
was written. ' 

Many personH toiiiiibutcd to tiiis volumu. rhc author wonltf 
like especially to thank Bill Ta«h for isis earlier contributions and 
his comments on this version; the inimy siudents in classes 
conducted at Michigan State University the University of Utah 
and The Ohio State University whose . icstlons and reactions 
helped to clarify the presentation- and the many reviewers of the 
drafts of this and the previous version -Paul Barton, Thoma". 
Bruenlng, C. Gregory Bi*^?„ Jtvhn Chcftop. SteVen Director, 
Ronald Ehrenberg. m Oi k bsrg. A^'dfcy Freedman, Patrivna 
Or«me, Frank Levifis, C.anb Mftngum, Heibert Parr.es, Edv^f -d 
Prescoit Cjoaia Rflhr*. Blowan Rcuihnacr, Harold •Sheppfiro, 
Frank' Shuler. mt^^i,^ Sis^iiler. Ernst ^f.romsdoffe, Ralph 
Waikfr, Barbara Wdftste.Ji, .md ,^Jfr;d ^uck, 

l\m book is ikdicdtflQ lo the «vj';mory <5r three men who 
introauccd me to fh« !r;v4.ifa:iQ < O' wdal programs: E. Wight 
Bakke, Joseph Boius, mi C!wf •IJ O Somcrs. 

, Michael E. Bor;!5 

Colimhm, Ohio 
April i979 
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Chapter 1 
INTRODUCTION 



Esdmtmn h ite %yiitmmc g^!henng of nHUon in ords^f 
to mak^ cbotes among altefMUve coui!#^ of aon. In tcfms of 
iod«l ^Qprnm, fvaluaiion ctn h€ MM upon m the ihttd step tn 
tte ptocci^ of prep^m itnpkrntrii-iH pj) Tli-^ TuhI 

sttp iJi^plM^ifli. Ptorfflung com\%u of ihe defirtJUon of the sockJ 
pf oWoti^ io^bt attacKed ajnd iha choice of ihe coarse of action id 
We takea In ibf soluilon of i^c pfobletns. The s^ond siq? is 
program op^aiioim, TOs it an auempt io solve the problaii 
tiffO^ the proyar^juUincd in tte plin. The iWrd si^ fi 
fviluation whJeb letkilo dtieiTnine ^bethef iht ptopzm wm 
fyc«^^fU} in sofving the pfoblern or cmjfd h€ ff^Of^ nuccf^ul fhin 



.^Rniiion of a 
- iodiJ ^dblffli $M 
dcvrio^CTl of 
apr^pm lo salve 
ibepfoblOT 



An auempi lo 
^Ive the sociaJ 
problTO ihf augh 

of nciion ^ 



Mi^Munng whether 
ibepf^ani 

sycc^i^Uy solved 

problem 
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Obviously. ihatmUnkn between c^■i>h^MM^ aiiu uic ut.ici ^ 
,tem to me psoom. PUflnln^ must be bwd on infarmyMon 
gaitofd in PM* mluAUOTU, Throu^ ih« Mudy of the ^uccc^a o 
ptelrMM we devetep be«« futuie mcap jo u^ohc «u 

to be *iiw«ful. SimiUily. props/n operaiions be iUwrti tti 
mwlmke tbe su^iful ckmenw of tr»* momm and to remove a • 
chan«. thw* fMWJ whitb mt un*ucc?«ful tn Uu'. way. 
mluwion permiui cOTmnUy to impfove md up^adc >be 
ciiwlHy and niix or- ofop«m^ by pro^ni^ng fecdosc* 
suec»-S€S and WlurB ■ 



<>rjc t«wn for toniiuain^ cv-Uij-r: /;; 
MW. p ft iiiink.ng. «Uon3l in,rr..l -.n '^^^^ 

fminimSie^ his cwsa) m hn pr«etit and {uiufi- agu*fffs 
fn irrm of s^ckl propams. how^vet , "^"^"^ J^^; 

ai»d «*« Jt^it«*loo.^ rt»« w^M Fflctors *!» have tof«^ 
^p^^mn 10 justify i*>c coaii^aanonor 

l^ytial IWUitS^ rilWJA^. ..u.^r^... 1 



pfOyam-plMnmng-budgaUng system 
. f \ UA^^^i&tf^ flw^saify under wlier budgeting 

fe ; but MB be 

t^vSaofc^^^ift. the .programs' . 

;*^H0w:^did the./ 

f ^^^^ to^icti » j^QiTi^n is 

ff^^: ■ ; sidi tte pteto^ il ^^to aptoii . to pian for : 

jfjp^glii^ oriitited-r-It tas!$ whether the jrfa lar bMi " 

. 4 ihft pfe must ^ 

Itofl^Sw^^i^ ori^|0 have su^ss.* It is ^peraUonJ clHmM^ b 
^^ pfopam and what- tlie 

ft^l^s fr^Iyatioe fergins with the aAiunipuon that in order to 
sM'^asprobte must ««ur.* Therefore, 

one of proce$s evalOTticn (referrFd t n m rfin/ro/^ valuation) 
to t^t for Ae these factors. For example^ itaie ^ 

, i. TOs' is i«.it €^ of Buw taaowroi^ /or ^fl^M the tl^P^ cvsl^tlen. toi ; 

3. Is ©f pfi3^l^ i^mt. ^'int io knf?* ■ 'Wa^ the >4M (the pto) lWiS| of 

*fii li tb§ cwito of iht tot" F^^i^i cii^uiitoiv mk% iQ mmnm ^h^bcr Us^ 
^t^oii^ WW ©QRie* wto ibm^ thai the p]m mpvmn$-^^ uttete 

* . m^^fk^ m^iM^^m tlmry. Tfm mUMllf took m mp^^^ ^umion, W)^ §hmld 

im ^ir^^pm ^ >i<>w my€h^" - 



e4uMtibnai; minorities usually reqjure a bftchelors degree and 
sp&mitraliinrf^r'teftcfettre cwtined to t«ch tn the public 
Mhools/ School dyificts, ire then examined to insure that their 
teadiefs art xkfdfled. There Is in this evaluation the impiiat 
aauaptiott that itt ofder for youth to l«rn, their teachers mujt 
m«wt state r^u^ents/Anotijer etainple would be the 
requirement that the participants in manpowet^toframs be 
unemployed, nnderemptoy^, or economicaUy disadvwitagcd. 
The assumption is made that in order for individuals m ihwe 
categorie to beBcfit from training and other emplpymeni^related 
propams, th^ must be the persons who receive the services. The 
control type of prmiws evaluation examines whether the in^uti to 
the pTOgram ffiwl a set of prjSdeicrmitt^ 

A swond type of proc«s evaluation is monitormg. The 
emphasis here is on m«Ung pr^etirmincd siandards; however, 
monitorini focus« on accomplishments or outputs rather than 
inpuu. Monitoring would ask suc^quMiions as "Has a youngster 
increase his reading level by a full year during the time he was in 
. the third grade?'* or ♦*Have the parilcipants in employment and 
training, progranis found jobs and increased their incomes?" 

Impact evaluation s«ks to measure the tels of the program. 
It tri« to answer the question "What difference has the program 
made?" The emphasis is on the changes brought about by the 
cxistetice of the propam. Impact cvaluatlDn seeks to compare 
what occtjrs, given the exisience of the social program, to what 
would ha^e occurred if the program did nqt exist. For exa^^pje. 
impact evaluaiion to deiermirie how much better a child 
readrafter a year of remedial tutoring in reading as compared to 
what his" reading level wotild Iwvc been if he had been left m his 
regular dissroom for thai period- It is impoftant to note that we 
are m^uiing the change caused by the remedial tutoring. We 
wish to know not only how much progress.occurred but also 
whether this was more than would have taken place had there been 
no tutoring. In the case of employment-related programs^ we 
would want to know if the incom:e of a pahicipani was higher 
during or after the program, than it would have been had he not 
entered the program. 



' . . .5 

It should also be noted that impatt ^valuation reaks to measure 
aU of the differences caused by the program and is not hmited to ^ 
only those which were originally listed as goals for the prggrwi in 
its plan* It IS quite r^sdble that social programs will have 
outcome other than those oHginally planned. For instance, it has 
been found that while increasimi education leads, as expected, to 
more productive population^ anfi more satisfied ^nd capable 
citizenry, it also is associated with lower fertility rates. 

Finally, strategic analysis seeks to compare alternative programs 
in order to judge their relative efficiency at a^ompUshing 
lonj-run/ large-scale outcomes. Such aTialysis might compare the 
ultimate effect on the productivity of the society of an increase m 
expenditures for^ primary education and manpower training. 
5trategic analysis compares th€ results of impact evaluatio ns for 
, more than one program. 

In addition to the type of quesiion being asked, another feature 
which dfttinguishes the types of evaluation is the time period ftor 
which the prograin is 'evaluated. Control evaluation axamin&s the 
structure of the prograni and its inputs-, consequently, it looks at 
information which is gathered during 'the period of program 
operation. Monitoring also takes place during th« operation of the 
program, but it usually will also gather- infofmation about 
'post-program results. Impact evaluation requires a longer period 
of time after the conclusion of a prbgranf in order to determine 
what the long-run results are. Strategic analysis compares the 
results of impact evaluations for a varie'.y of competirtg programs 
and must wait until all of the impact evaluations have b«n 
conducted. 

The dislinctions between the various types of evalu^tioii may be 
clearer if we use an example from the medical neld;. : 

' h A child is born. We check that the nursery is clean and well 
staffed by trained personnel, and thai the child is fed on^a 
regular schedule. ^ • 

This is control evaluation. It examines inputs to the process and 
• measures them against standards of necessiiy. 



2 As At chUdpows oldtt, we.^odic^y measure its height and 

- weight. We »lso see if m chUd wdks by the tifte he or she is 

'two ywrs old. ' j , ' 

TWs is mo^torini. It Mamlnw progress ai^ output against 
predetennta^ s^dMds of what should ocmir. 

3 The cUld is inaoculated against various disases. We wish to 
" m^ure whether the imioculationf has led to a loqger and more 

productiye^Ufe. ' 

. This is im^et evaluation. It reiates changes, in outputs to a 
, chuigd ia^Ae inputs. 

4 Ai evaluator attempts to determine if an expenditure on public 
■ health suasion, primary mediMl care, or medical research is 

the most 'efficispt way of ^extending the longevity of the 
population, ' ". / 

This is strategic analystf. It compares the efficiency of 

- ^timative strategies for accomplishing a goal. 

WhoETaIttitM7 

Types of evaluation may -also be differentiated by the kinds of 
aienaes that conduct them and the .uses to which they are put. 
Since control evaluations basicaUy deal with whether prescnbed 
conditions exist or do not em, it is the easiest type of evaluation 
to conduct tnd does not r^uire an intimate knowledge of the 

• prowam's qperations. Thus, it an be conducted by individuals 

• outside of the program. usuaUy bf accountants and auditors. This 
type of evaluation is often associated with the.Generai Accounting 
Office (OAO), the "F^eral Representative" who raritws • 
prOffams,* tod*othfr "outsiders" who usually attempt to asw 
ironams qtfickly. On the other hand, monitoring usuaUy involves 
the propam operators or members of the program s adpimstra- 

^Uve sWff sinw one of the basic purposes of momtormg is to 
provide the propam operator whhearly fwdback on his succ^s 
^d "failur«. Monitoring questions are often built into Ae 

Ttropaffl' s'management information system. 



" Impm evaluation is usually done by an extemfl apncy. Most 
prograin c^rttors do not have the skills, budget, time, or interest 
nec^s^ to conduct impact evaluations. Further^ internal 
ev^ualions may ^ve the appewance of conflict of Interest, Hence, 
cqntratts for' such ev^uations are often ler^by federal agencies to 
p^vate researchers Qt universities, 

S^tepc analysis can be conducted only at the highest levels 
since jit compares propMns that ciit across nprm^ agency and 
jims^ctiffnid Un^, For instance, in the^federal e^ceeutiye br^h, 
tHe Office of MMagraient and Budget (0MB) perform^^ 
func^oh. In Confess, the House and Senate Budget Conwwttees 
are Eispottsible for ^ocatinj funds across broad subject areas, 
while the J Appropriations Commitlees divide the funds among 
com^mtog programs. The Congre^ional Budget Office (CBO) is 
changed with providing the anijysei on wbteh these decisions ar^ 
made«^ ^ * ' ^ 

Orga^zatioQ of This Primer ! ^ 

Tins primer is concerned with impact evaluation; ptocess 
evaluation is already covered well in other works,* and we shall 
not cover stratepc^anriysls Since our purpose is to introduce the 
reader to the b^ics of evaluation^ techniques. In addition, the; 
prfmer concentrates on questions dealing vrfth the mechanics^ of 
cohducting eviduations and oniits most of the theoretical 
discusSions^ which ihave occurred in the field of evaluation 
(^though References to these discussions are provided). While the 
techniques for evaluating social programs are not difficult, most 
past evaluations have not included all of thoie basic components 
^cessary to arrive at reliable policy decisions. By giving special 

' 5, The typ^ ©f tviduitiofl are oBviouiiy not^muiually exclusive. As noted earlier, in the 
of 1 failure (dettrmiii^ by anlm^et evaiuation) wa want to know? whether the theo!;y 
of the plan or its ^emitipn was faulty (which requires proeess evaluation). If we have a 
succ^s (again udng an impact evaluation), we will want to make sure that the success was 
due to the plan being foUow^ and not to^fome deviation from the plan, 
i 6. See in particular Foms on Manpower Pianning (16), Menii (21)/ and%.i. 
i Department of Labor (34). 



emphasis to th4 areas in which past studies have fwledj we hope to 
^rwent an elementary discussion which will help correct the 
appiffent defiaehclw in those past studies. 

'^Readers can best use this , primer if they attempt td test their 
knowledge and make use of it as they go along. To encourage this 
apprdacti,.exercises we Included at the end of each chapter. We 
also suggest that readers examine several of the studies cited in 
each sMtlon... Each of these' was selected because it presented 

• theoretic^ argMienft in greater detail than was possible in the' 
space avdlable, popefully, this'pfimer, togitherwith the exercises 
and references, will serye as a jumping-off point in the evaluative 
ptocm, pernutting evaluatdrs to develop their own analyses 

^Mthout committing the same mistakes that have marred earlier 
studies^. . 

Chapter OutUne ^. ^ 

The purpose, of the evaluation process is to provide policy 
makers with the basic datw necessary for them to make decisions 
wisely. Impact evaluatiohs of social programs examine the 
long-run outcdmetand' view^ success and faihire in these terms, 
t^ey should provide five essential sets of information. First, they„ 
should provide the data necessary to determine if a particular 
program should be contirtifed. ' Second, they should determine 
which k alternative programs achieve the greatest gains for a 
given cost. Third, evaiuation^hould present information on the 
components of each programed the mixes of eomponents which 
we most effective for a- given Expenditure so that maximum 
operating efficiency can be achieved. Fourth, evaluations should 
provide the first three types of information fot paTtidpants mth 
different characteristics so that a 'decision maker may determine- 
which individuids we best served by each program., Finally, in the 
' course of evaluating existing programs, data should be gathered' 
which will suggest new methods for attacking social problems. 
F%w iinpact evaluations of social programs have provided all of ; 
this infortnation to date. 



' . One of the'major problems in the evaluation of employment- 
related social programs is that these programs encdmpass a wide 
variety of desired outcomes for the nation's workers and potential 
workers. QeneraUy, they seek to improve the eqiployment 
situation of the program clients, and irt this way to better their 
economic,, physiol, and mental well-being. The programs also 
stek to increaste the productive ability of the nation's human 
resources' and to reduce poverty and social dependency. These 
gods, however, are broad and difflcult tq operationalize. As a 
consequence j evaluations of employment-related ^ograms often 
have been narrow in focus, usually limited to the ipost obvious 
effects of the programs such as the average increment in earnings 
of participants o^ the number of participants placed; in 
trainirig-related jobs. Other less apparent but Wssibly important 
impacts haVe frequently been ignored. Therefore, our first task is 
to define more of the basic objectiyes and impacts of employment- 
related programs and attempt to establish critipria to measure 
=these. ,' ' . I ■ 

Other problems which have arisen in social impact program 
evaluations revolve ^around the question "Whom do employment- 
related programs affect?" It appears that past studies havi 

.■ excluded, often l«cause of a lack of data, many persons whose 
labor market sx'per^ence was influenced by such programs. In 

' Chapter 2 we poifli out some of the other groups that should also 
be examined. 

The designs for measuring the success of employment-related 
programs in past studies have often been weak. Many of the 
problems have arisen in the use of control or^comparisdn groups. 
To estimate the effects of a program, it is necessary to compare the 
experience of the program participants with that of some reference 
group whose .experience can be said to represent what would have 
happened to the participants in the, absence of the program. 
Unfortunately, results of past studies which were contrary to the 
prejudices of policy makers have too often beei^'dismissed^on the 
grounds o°f noncomparability between progranx. participants and 
"controls." In Chapter 3, we present a procfedurrf designed to 
solve the comparability problems, We also supply descriptive 
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mformaliou^n possible sour^ of data which might be useful for 
measuring i^e baieflti of employnient-relatedi prograaiS, 

'yfii pr^lems toVblv^ costs of social 

^offum Save b<^n simUir to those involved in measuring 

^rbffm silcc^r: Past studies often have not raeasured all of the 

^ppropriati c<ists, have inadequately selected control groups for 
co«^y««i» tod have igoored some groups who incur costs. 
Tliese itfe ^e iiiues disTO^ 

A fmai proW^ which has limited the usefulnesi of many 
previous evaluations has been the lack of compaf ability in the- 
pr^tation of the resupW these studies. Chapter 5 presents a 
suggest^ sch«na br comparing program bteneflts and; costs. We 
also suggest in that chapter t number of technic^ aspects which 
have, often bwn laclfflig in previotis studies—the use / of 
multivariate walysis to s^^ate the influents of the wide variety 
of possible determinants of prq|ram sufccess; the use of marginal 
toalysis whenever possible, antf the methods for projecting ifld 

' ^scounting future efferts of the pltgrims,^ Fin^y,.,we present a 
summary outUne of the procedur4Mteps disciisiedxih^ w^ 

■ chapters. ^> " ' , ;'r~^-"'"' 
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Itoerdie 1-1 ■ • J _ ' 

Lab«! each ql the foUowing types of evaluation questions as 
process (coatrol or monitoring), Impact, or strategic analysis. 

I 

1. Do all vocational rehabilitatipn coua^felors have degrees in 
counieling? , 

2. ' Does each student welder have 15 square feet of floor space? 

3. Are vocational education program graduates placed in jps 
which make use of the skills taught to them in school? ^ 

4. Are no more than 10 percent of regiitrants In a "holding 
status" at any point in time? 

5. Has a public service job increased the income of the perfon 
hired? , 

6. Does vocational, rehabilitation lead to happier, more satisned 
clients? 

7. "Is training or work experience a bttter way to increase employ- 
abilitv of youth? 
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Chapter 2 

E IMPACTS 
PRbGR^MS 



DeOitng the Goite ud Inpacu 
of Emptoytecat-itdit^ Prapanu 

Eniployment-j'elated social programs noay affect many persons 
and insiitutions in a variety of ways. Some of thwe are dir^ 
t«neBts wWch are planned. We deflne ihose effects which arc 
objs?tiv«jo/ the proram planners or operators « "pro^ra 
goals." there usually are many other jwssible effects of mwe 
propTOS wWch are not ariticlpated in the program plan dther 
because th^ are side effMts or occur to persons who are not 
directiy involved in the program. Some of these are posiuve 
6ut<»in^j others may do harm. Th«e unanUcipat^ effects. 



together with the propara goals, we c»ll ''Impacts 
showd in Ch^2.!, program goals are a subset of the program s 
impacts. ^ 

Chart M 

InipwMi of Em^oyniint-Rdated Progmms 



Goals 

negative 
oftitcomes 



■^oih?r 
positive 
■^^^ outcomes 



la our OftoioOt uatttm^ should bt roade lo mMsure all of the 
ImpMt and not just thi goals. Rationale for ihis opinion appws 

tUu^mm IstensM^lf ^mmm m Impacts 

^ In donsWfftag imwefs, it Is useful to distinguish W^wmn those 
which art ultimate obj^tiv^ mi those which are only 
imWrneduite stq^ to a^ehievdng those objectives. For instant, 
hl^er amual ^^olrip for^ftidpanu may ^ the ultimate goal 
for a pit^p^r Intttm^aie outcome which are somwitn^ 
oamined to m«sure this goal are incrws^ wage rat© at 
pla^menl and Incrouied number of persons plac^ In training 
relate jobs. The intOTi^iate ouiwni^ can be differentiate in 
two respwti: they ocwr swner and/of they' are only ptfljal 
measure of the impmci, , 

TW n^ for a time disUrcilon is obvious. It is usually 
.n^^itai^ by an inability to wait unlil all of the cons^uenc^ of 
pro^wi partldpition have occuned. fFhis is natural when 
programs am affect the entife lifetime of pwUcipants.) There is a 
dangi^, whieh wiU be discussed further in chapter that «rly 
r^ults may not be indicative of longer term cons^uences. Shortly 
after the propam, the pmieipants may find themselves at a 
disadvantage if they have ^n taken out of ihe labor market for a 
iubstanttal period. On the oiher hand, ihey may have gain^ a 
short-run , advantage by making special contacts during Uie 
•43r^tun. ^ ' 

More seriomlyt intermediate outcome may not mwure the 
^..^^ t^r*nr^ *hf* MiftfHfif*^ ^^h*#vHiv^ Fnf tn^tftnce, f he absenc^ of 
higher wage mies may not mean annual wninp j« not higbcr 
since the program may IncrMS^ hours of employmtnt. likewise, 
persons may be placed in the jobs for which they were trained but 
' these Jobs may jmy less than other jobs which might have bmn 
held. In Uiese oas^ the implicit assumptions that the Interm^iate 
outcome were identicai or highly cortelatcd with the ultimate 
objective not correcL ' 



Pot am^ rm^^ it h owfiU to dtflm imj^cts in t^tm of 
ttlriaattt objfirtto wl^vo' pottiblt. Siroi^ly* m$ shouJd be 
lakia wfcffi pt^Ui^^ m^^mai ^iicna to use jhor^ -^hfch 
mc^ do^y r^n^ t^ laamtit objective. Tht funhcf r^movdi 
wjtol h h^s4 mmsm^ h tmm the ^cniuel out^me* in conctpi 
and in tiTOt to pnt^ wUl be the diame of inadtqutitc or 
improp^ ^rdmatk of tbe mie Impacti. 

Somm oC InfMiM IK^Mon 

The fijst of4m of btttintis in conductini m impact evjJuaiion 
lo dfflM tJje brMd toptrts and the more spedfic critCTia which 
tmy be us^ to judie the ef fertiven^ of the pi opanw Wfttrt do 
we find th^ Im^jm md crit^? 

TlNi obvious p!a« to be^^ii is with the ipmJs m defm^ in the 
le^slaliori of e^ablUhrf pfopanis i© m what ihe diaftefs 
thought the projiM would ai^omplish, Unfortunitely^ ihis very 
often leads to staieoirats thak^ are difficuh to op^ationalk4 wiih 
^icrla. For Imimm, the CoropNh«r^ive Employm^ii mi 
Training Art of 1973 (CETA) had the fbilowiBf Siatw fni of 
Purpdset 

Ii is the pu^se of this Act to provide job uainmg and 
emplo^rat opponunitl^ Jor economiraily dhmi 
vaiitaged, unOTpIoy^, and underemployfd persons, 
and 10 assure that ifdning and other servloe* ted to 
majdmum OTployroenl opportunititi and enhanee self- 
suffidency by tstablbhing a flewble and d^wnuSlz^ 
^tem of FedfTtJ. State* and Imxl prograjras. 

This statement d^it^ the organitauon oi ihe proffrain 
(d^mifaUi^)* the iyp€ of fwrticipants (unCTployi^i underem- 
ployed* and ^nomi^y dtodvantaged persons), md ^t iypm 
of senriw tp be perfoiimdl (job training and employinent), but 
vague as to the outcome to be achieved CimaKimum erapIop«eni 
opportuttlii^ and enhanced sel^sufnciency). A review of a few 
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oihn -piecw of lefrfstaUoo will mdimii that tbiA luk .4 deafly 
mtM teflslawve soal* »s not the excepiion, " 

, •■ . .. ... ' 

le^slaiive fittiifl^ hsld prior lo the «»ab!«hniem of she progr wn 
and ihe hwlnw beld on oppropnauonA, A«aifl, howem, om is 
mum teft ^lih vsfue jiaiemenw of o^'emU objecHves. Mofwver . 
ihe goals of a pro^ftm may ctmoge over lime from iho« orl^na^y 
siaied in ihf Ic^iiauve pf<K«*s. For msiwce. ihe Manpower 
Devetopmem aad Tfalaioi Act of 1962 was originally passed w 
combat dhplaameoi caused by auiomaiJon. bui s«v'«^ yeaf* 
lata 11 was ujwd prinwHy as a program for mm^mt. mmmny 
groups Imo the labor-fofit wmmi tMtm^m Icftsfauon. 

One win also turn io the op«fatofs of i.ne pfogfam. p/opain 
diems. and oihct poiwuiai umiQtm evaluation to ofciain jhejr 
pcrccpilons of 5hc programs gMis ?• iwiwrianf to note, 
hovwcvc^. that fhc goifls of progfom cHwts ?« is««ly to be m^iy 
Individualued and ISmitcd to their, pcrwnal desires, Li^ww, 
program manasers oftetj think mom about m itmm v.h,ch ibey 
. jnust prmtdfi and the iwhniques for ptovidmg ihew w» mv#s iMo 
about ihe impacis of providing ihes^ «(^?c« Thtfcfore, ilte 
- lift,. t. «ff»»r sncnmplwearid najro* in 

iheif viewpoint , 

The agency fund^ag jhctfvaiuiino^ slwrnfty Qiter fscommenda* 
lions on which poicntiai mpmmm corndtt. A problej wiih 
relying on Uu4 imtm to defmc ihc ob|Wtives m m siudjed s tbe 
poMibiUty of bdng wopicd, driven 09} ar«ue* for "gtwl-free 
evaiuauon" whe/c the cvaluaioi «lKi< sft? crisctio tor c^»luaiin« 
a progiaiti complMdy indepecdentfy the P^o»'**" ^Jf''^; X. 
(ndepcndcnce, however, may w^y * « *i»iwi*iMi ;w w . J"- 
goals and CTiteria, Know ihe goaH of iHc ^taf! mt4 m\ 
nec#*mrily bias the c^aluatoi 

,rt*34C ot 4iwi«*^»<t.J. ..^.i^f'rTrf. -I n->f»wr onv 
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wwi^Qt'^Sk attaf iJ^ to^s I© b#aam^, it is to^tett 
m Wm w tWflk of iU iBi p<wibl€ trr« 10 ^blch li^t pto^^ 

snt^ obl^?^ for tit^ pr^ria Is ml taj^rwot. For €3i^pl#, 
Oi^. is pWhWi III m CCTA l^arieii which t^dj^^ tte 
^^prsa^i tflP^ on mmUm t^ltb. CETA My tevfe m 

tara^ CTOtwi %ritb sfriid^; of At 

aMitlooml N ^pni on bWlUi setidc^ If ih^t popim^ 

is fu^mful to fi^^ vfQtkm inmiim tnd 3)^ lyfeiMUil 
portion of ihe uN|mta$ uodtf tb^ Aat ^ilt; be m aica of b^ift 
5^^. Thus, whllf hwlih w< 00! menilened by ihf auihors ^^f 
iht It^rri^l W much afJacted by CETA. Th^ ta^cf ^ 
this mm inay bt »Midct# vifry icnpa^iini by wmt pmpl^ ump$ 
ihe pfQpmCs eV^uaiUorts for ^c^sloft making: ' ^ 

As a ^icft! mtei u'is much beti^cr 10 aiiOTpi ,w ram^itft 
impiils whkh prove not lo ihan 10 Ignoft fmjrarts ^hich do 
wis*. There art two rtajons for ihte Fii^^ boih ihf poUilial Md 
.^aomi0 s€tne may cbMfe u Um^.^«i, and iht goa^ of iht 
pfopifa may change m was with iht ManpoiAfW 

I^dspmeiii and Tfiiiiilng Ad of I MI If mw g^Jf wett nof/ 
iiiduded in ihe mJyationt ii may now^ woithl^s since ii >i4U noi 
aMv^tr the qy^uonf being asked ^ndrr iht new iiondJtiomr 
StCTttd, sl^ modeb of the latK5f maikei we vety in^mpliMi. 
ihty m uisabte io pr^ict with cciiaifity whiere iJte topKi or 
iiWlow«i(Jo^erti-rdiii^ pfogrtiB^ wUI be fetiv SodiJ sdratifw 
aift ujmbte 10 010^ aff of the retefioRjhliH or 10 Mie whit the 
effect of a ':*wlfrakf action wiil te. As more ajnd more tviMuitiOM 

waMiirt, ihe models will undoabicdjy impr ave. 

Thi^j whfle ii U valutblt to dttetmint ^ho wiU be iMtoi ihe 
evaJyirlon and whil iU u«i ait to frf . ptMi w^mplctc iffiS|tfiU on 
the f t^ms of tte pfO^ruM undrt wajyauoa usually iMi^ U 
lin^atlve to go b^oad considerations 10 Imlude aU of 
thf pmdbk Impacii* Orat wt shotjfd be takta befoft an 
objtoivt U eUiniMi^ from con^dtTauon due to ihf j^wiai 



iw^rteu W now iJ^t ^ ^acmtfy four 

mtmtm goals ^ iat«n. the objt«iv« of " 

changes. ExpP'» ^ ^r.^.^ ,^1 ovefaSI d»mb«non oi mmm. 

WhU earww m4 ^f^^^^^l^, ,f own imcf «sw. Wot 

- h« win «n«.n- ;^r^,^..^.n, 

label !bfc« ha* ma..^.^ . i^- ^ ■ ^ , ^-a, 

^SS;;;----^^ ........... 
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uuwsied in aggregate change^ m well as ihose dirtcUy affuciing 
them. Likewise, the effect of programs on individuals and 
employers ^il! determine in pEji the prograrns' success in terms of 
socieiy. Incraised empjoymeni of panicfpaBis in progniiiis is 
likely to Ijonprove aggref^ate employTtient, and improvement in the 
production of individual firms niay ieiid !o increased aggfcgatc 
production. 

There also may bt some overtop among the gOBh of each of the 
pmum. For imiBmt, the feduetion in an individuar^ unempioy^ 
mmi may mcxm$c his mnm$% m well as decrease his feeling oi 

im the indi^dumi, we Miw that ali^hould be considered. 

Thete may be connicu, howevc^r, among ?he gOBi^ of the 
differefii pmim md among various goals for b panicuiar pany. 
Thus, we tmyfifid thtia prpgnini which improves !he income of 
fhepiirndpanfs is very cosfty to 0\e government or that a program 
which IS highly efneieiii at incrai^ing aggrepie production lm6$ 
togreaier tmquny in iht nauonai disiribuuon of inconie. Thmu 
cmflku of po^-^iblc prograin achievements faj>c the problem of" 
fanking ihe obiKUv^ of tmh oi i\n punm and of determining 
which party*s goals have prc«dertce. While on a th wetiMJI level 
one can argue, tot f^ocietal objecti^^^^ should be paramount, 'Ute 
^vtHiaior muse be ii rcalki He should recognise thai *e rewards 
mi .4)iU of cmplovinuni- Filiated ;fu^g;aiTi^. w pmkulm :fner>sted 
pardc^ may pUy m mporimt ti^\c m ddermimng the sa^, scope, 
and even ihe mumm of the progrMv For insfance, stnc« ihm 
p^ogftini'^ typicaliy v^^lHfMary, if proff^m^ which -hB- ' gregi 
poieiniiiJ ^oqii! and govefniTieni benentA do not provide surfiaem 

the ^nelitj. for %<Kitiy of gavcnmien? ^vill not be rr.iiii7?d. 

Ukewis^, ^.houlA noted Unu ui^--^ fife oihcr proupn, 
piyiicuiaily pre^ur^groupf;, who nisy t^c !ch dirctily aff^ted b^^ 

^^:ay^ of their poliuca! mtlyence Social pragfani^: m 
govemmerit piogrum^ munc ^^miy pahUcm demands m m 
imompmhm$€Konom\c md ^xiui jiu^^Hi W1;l:tc the ptogiMns are" 
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l^tsd, who partidpates, and who manages the programs may be 
Sportant poUtiSly a* what they do. Thus, the ef«^ of 
^^cnr.rlt«l programs through th«r ^^^^^ 
impaci upon voters or wmpaign contnbuiors may be .xueroelj 
important to thdr poUtiwl suTOval. 

Even if we imore these poUtical consid«aiioiw cinploymc^U 
rJSp ^ lndir«tly affect a number of p^pk. ^r 

nsM« eSyc« wiU fa« a, dif Cerent labor supply and 
noSctoS Idifferem demar^d for their servnces ,f a-program 
^sSw^trtihing large numbers of workers f<^ a paruc^^ 
^^^on. tJmon, might fa« a variety of changes if ^^P^ 
Sees a number of workers from mm^iy pmiP^ 
backgrounds previously unrepresenttd m the plant. The^f«^^« 
w Spanu may be adversely aff^ted by the costs which they 
, hlv"^^ ie the participant is in the program but m^y 
Sr^f^e program successfully incrwses famrly ^ncome^To 
■ to disSSon. one should consider the «tegones oj 

S^i^w^poaibly could be affected by s«ifl P'OS'*"^."^^ 
many ^sin which ih«< programs coUld have an impact. 

Ltotlaro' 

To fadlitate the choice of impacts to be studied, we pr^ent lists 
of^^S benefits for society, indi^duals, employers, a^ 
govS^- we feel that all employtnent-related P^o^^^ 
fuSn of tb«e impacts, but simuU^eously 
Sch proyam \vill have a different method f ^» 
StSd WiU put a somewhat dif f™p^« on 
them Further, bemuse it is our hope and belief «hat the^smve 

negaUve impacts which may occur. 6ur d;*^)^^^"^""^" " 
«^ in terms of benerits here and throughout this book. FmaUy, 
^St U ^^sly not alUinclusive. It should, however, provi^ 
of the . most imponani outcomes of emplo>Tnent.re lated 
^^s M™w each^nefit we present operational cntena to 
h^«e« of an employmem-related program m m«trns 
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the objective. These criteria are presinted as examples of jhe 
measure which could be used. Again, the list is not meant to be 
^Wnclusive, 

A, Benihts FOR SwrnTY ^ J 

1. Improved Equity in the^'l^tnbution of Income and 
Emplojment, Espedaliy for Target Groups.^ 

a. Increased Incomes. The incTcaie in the income of target 
group members relative to some stated goal such as the 
average income for all workers^ 

b. ' Increased Employment, Tlie increase in the percentage 
of time in wWch all target group members are employed 
and the d^rease in the percentage of time in which they 
are unemployed after the program, relative to the . 
average fo^all workers, A less useful meafure because 
of seasonalhy and time trends would be the increase in 
the perc^niaie of target group members ^who are 
employed at given times relative to a stated goah 

2. Increas^ National' Production, 

The increasr 1 1 the Gross Nationid Product (GNP) which 
should approximate the sum of the changes in, earnings of 
all arsons affected by the program, including persons who 
are not program participants, 

3* Reduced Unemployment, ^ > ^ ' 

The d^line in the average percentage of time in which 
persons 'affected by the program, including nonparties 
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3. pfWips which mi^t N eonjidered are: 
8. Persons defined ti wnoml^y disadvantmgfd. 

b Mralm Of minorititi (llaeks, Nilive Ameri^ ' 

c, TTie handiapped (physiwily impaired, menialiy retard^, mtntaUy UK aleohol and 

substaaee sbuwi), j u \ 

d/Gtm^ with high uf^mployment nm Oeenaiers, the aged, ^^effendiri, lehooi 

dropouts)* ' f 

Of^^ weiving govemment benefiu (weifare retipienu, unemploymeni imuranc^ 

elaiMflti. vfttraui). . 
f. Olh^ (f^l€ ef-households* farnieri, persons in deprmed fam, tfte 
iiamp1o?^» and the undeTemployed), 
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ipants. are unemployed after the program. A less useful 
measure because of seasonality and time trend would be 
the change in the percentage of these persons who are 
unemployed at given times. 
, Increased Social Satisfaction. 
A. Increased Satisfaction with Social Institutions and 
Increased Sodal Participation. The increase in partici- 
pation in political activities of persons affected by the 
programs. The improvement 'in the average scores on , 
scales of attitudes tdward social institutions, such as 
schdols, police, politics, and welfare agencies." 

b. Increased Job Satisfaction. The improvement in average 
scores on job satisfaction scales.' 

c. Increased Overall Satisfaction. The increase in average 
stores on social indicators.* 

i. ^tabie Prices. 
The stabUity of wages and prices in those industries and 
occupations in which persons affected by the program are 
employed relative to average changes for all wages and 
prices.. Special attention should be given to "bottleneck 
industries and occupations. > 

J. Reduced Antisocial Behavior. 
The reduction in the number of persons affected by the 
program who are arrested and convicted of crimes, who 
participate in riots, or who are involved in other socially 
unacceptable activities. Reduced recidivism rates and 
parole revocations for former inmates of correctional 
institutions might also be utilized. 



4. Avdu,5eof pdUtiMl sMles should be coniuUed such as John P. Robinson, Jerold O. 
Rusk, and Kendra B. Head (26). ' • a 

5. For a eompendium of such sMles see John P. Robinson, Robert Athmasiou. and 
ICendra B. Head (24^. 

6. Fflr thwe measures see Jdhn P.Robinion and Phillip R. Shaver (2S). 
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7. Reduced Dependancy o^\Goyarnmint. 

TW reduction in the number of persons who receive public 
assistance and unemployment insurance, the amount of 
each received, and the proportion of time these are 
received^ Psychological scaJes of dependency might also be 
used to examine the degree of dependency as perceived by 
persons whom the program affects* 

8. Increased Voluntary Leisure. 

The reduction in the number of hours worked to conform 
with individual desires, A second measure would be the 
reduction in the proportion of persons affected by the 
, program who work more than they desire. Finally, the 
improvement in scores on an attitude scale measuring 
satisfaction with leisure might be examined, ■ 

9. Improved Family Life, 

The reduction in the proportion of program-affected 
persons whose family lives ^re negatively altered (through 
divorce or desertion), Chanles in attitudes toward other 
family members could also be examined, 

10. Reduced Discrimination and Improved Race Relations, 

The proportion^ of persons affected by the program \yho 
improve their j5ehavior toward pCTsons of another race, 
ethWc group, age, and sex. 

11 . Improved Health. 

The average improvement in the nutritional level as^ 
nieasured by changes in the amount of food consumed and 
^ its protein content. The effect on health can be measured 
by the reduction in avisrage number of days sick, the 
proportion of program-involved persons with emotional 
problems, and the value of health services needed by 
program participants. / 



12. Improved Housing. 

The average improvement in the quiUty of housing based 
. on the Census deflnitipns. 

B]^ BBNBmrS lORlNDIVmUALS 

\ 1 . increased Incomes, | 

^ the average increase in the ii^omes of participants. The 
incoine increase could be frtm either increased employ- 
ment or hi^er levels of pfoductivity. Separate calculations 
'may be made for various groups of participants. 

2. Reducecl Unemployment. 

For various types of program participantsrthe reduction in 
the average percentage of the tide after the program that 
they are unemployed, A less useful measure btcaule of 
seasonality tod time trends would be the reduction in the 
percentage of different types of participants who are 
unemployed at a given time. 

3. Increased Satisfaction., 

SL. Increased Sa^ctim with Work. sy^ 

provement in scores on job satisfaction tests by different 

types of program participants, 
b. IncKosed Satisfaction with General Conditions. The 

increase in average scores on social in^cators by 

different types of participants. 

4. Increased Social Status, 

the improvement in social and occupational status of 
participants with differing characteristics as measured by 
socioeconomic scales, 
"5, Increased Voluntary Leisure. 

The increase in the' average number of hours when work is 
not sought of desired at the going wage for different 
groups of persons affected by the program. A second 
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maasure would be the reduction in the proportion of the 
groups who work more than they desire. 

6. Reduced Dependency, 

The reduced proportion of different participant groups 
who receive public assistanci and unemployment insurance 
and the reduction in the amount of each received. The 
reduction in the degree of dependency as perceived by each 
group could also be examined. Scales of dependency might 
be used. ^ 

7. Improved Health, 

. The average Improvement in the nutritional level of 
different types of participants as measured by changes in 
the amount of food consumed and the protein content in 
their diet. The effects on health can be measured for 
' ' different groups of program participants by the reductions 
in their average number of days sick, tfie proportion with 
emotional problems, and the value of health services 
provided to them* ^ 

8. Improved Family Life, . 

The reduction in the propQrtion of program-affected 
gersons whose family lives are negatively altered. Changes 
in attitudes toward other family members could also be 
examined* 

9. Improved Housing. 

. The average increase in qualiiy of housing of program 
participants with different characteristics based on the 
Census definitions, 

C. BiNBFrrs for Eotloyiks 

1. Jobs of Specific Employers Filled. ^ ^ 

The proportion of participants accepting jobs in "bottle- 
neck" industries, in occupations where workers ye in 
short supply, and with particular employers. The number 
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of v»ca„ci« »nd public En.P>°ym=«/;™« j^^^^t' 
nUed, by indmuy and occupation, arc a s«ona m 

2 Jobs in Particular Areas Filled. ' 

The number and proportion of participants who find 
irpCrinlabo^shortaBea^^^^^^^^^^^ 

::&b"B;;rot;":^^^^^ 

areas could also wrve as a measure. 

3 Improved ProductivUy of Particulai 
rmployers' tabor Forces. 
The increase in av«a,e output per houlr worke^^^^^^^ 

examined, ' 
D. BBNims FOR QovBRNMiNT Operations 

1 Reduced Costs of Qovernnfcnt Operaltons. , 

' =e:?^5anjer^P----^^^^ 

servi,ces. ; • 
• 2 Reduced Transfer Payments. 

S fluy members should also be measured. 



3, IncrcDsed Tax Revenues Through an Increased 1ax Buhc. 

The increase in the taxes paid by persons involved with the 
program. Separatt* calculations should be made for 
federal, state, and lo^a! taxes. Th<^ federal level should 
include persona! income, mem, pind social security taxcn, 
and local and stftte tax measurements should include 
mcomc, sales, and prcpcrty tuxcH^ 

4. Increased Number of Persons Available forMihtary 
Service or Other Public Service. ^ 

The increase in the proportioii of youth who are classified 
as acceptable for military service. Peace Corps, VISTA, or 
similar types of public servicf 

DlscuasloR of Criliria 

The actual measuremem of the impacts is a difficult but 
necessary job, Cons^uently, some discussion of the rationalf for 
those criteria appearing irCthe Hst above may be useful Because 
economic criteria are much better defined at this time than are 
some of the others and therefore easier to measyre, w^. wlU begin 
with them, * . 

Employrnent-related programs arc directed at improving the 
earnings of program parlidpants for at least thr^ reasons. Firsts 
as skills are improved the productive capability of society is 
increased (i,e,» ii& im^ovemem in skills of the labor force will 
permit society to ^produce more). In the 'various types of 
employment-related programs there arc several techniques used In 
the atten?pt lo iricrease ^productivity. The CBTA Title II, 
Vocational Education, Vocational Rehabilitaiton, and WIN 
programs provide skill trasnirtg an^^or basic cducaiion in an effort 
to increase the productive abiUties of participaiils. These prbgrami 
also contain components to increase marketabHiiy by, providing 
useful work habits and experience, they, along wlh the 
l^ploymeni Service, also pro^dde labor market infortnttion, 
including information on where jobs are available, in order that 
appliMnis may belter match their skills with the demands or them. 
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Next, the impact on eamlngi may affect the inionie 
distribuUon. Ajalo, hlf lier produeUvlty provided by cmploytneni- 
related prorW" «l>owW w higher eartilnp-ibc con«pi of 
InvestoentlnhufflanMpital. pependJflgon who the cedpienLs of 
thepropamsare, theanUagi distribuUon and, subw^uentJy. the 
Incoma distribUdoa may be altered. If. as is typical, the 
«fnings-inawtol wrvicea are provided primftrily to ?he poor, 
the aiitdbuiion of lucomc may be improvcu, 

FlnaJiy, jodety appears ifltcrcsted in iiriprovtng thp earninp und 
employment of parUcuJar group's lit *ht society because of tJie 
bclier in the ••Froieitant Eihtel'^ Society appears intent on 
replying welfare with worlt and placing a posiUve value on 
lneomeeaiti«i m opiw»^ to Incoroe from other «6uro« (at least 
araoni the poor). Iwofar aa aie empioytnenwc!aicd prograina arc 
dir^^n 'li^flc groups who are poor and likely to rc«ive 
transfCT paymeota. lift achievmTtent of higher ■ earalnp and 
employmeBt for them Li looked upon as a benefit of the propam. 
'-How a«! Uw« bensfitJ to be measured? In the m« of the 
chan^ in ttwieiy's production we assume it svlU be equaJ to 
increment ratw^ by the program in the productive abUlty of the 
propam parridpanu.' Since it is difricult to measure producUve 
ablHty dir^y, the evaluator must rely on marginal productlwty 
theory which say* that the incTcment in she ^arginal individuaJ** 
output is «|ual to the Incremeti'i in his wage, SMuming perfect 
competilion. Thus, we an say that employ^t-related programs 
indTMia the output of society in an amou^ equal to the Increment 
in the wmlnp of the program participanis, given certain 
assuinptions such as perfect knowledge and mobility and full 
employment.* This incremen! Includes the incrrase m total 
compensaUon-fringc benenis m well as wage or .^alftry paj-Tnenis, 



7. In ChipW S. sltrrMlive pinsibiiiiitt »iU ^ ath 
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whkh Meur»n« ihe progiih's tm(i^»^^m u bemfiu M M ^Am *htih «m uunm -Jm 



the value of fringe benefiij on an mamMnl bmn. &.k m cffon 
should b€ raade. We wunlly find ih&e fnngw increase ^^ith 
earnlnp a*^d are somewimt more subiUinuaj 4n union than in 
nonuiiion planif^ TheealcuiaUon of the maetnent m eaimnp abo 

before tMes and ahouJd include ib.^ concnbuiion^ by ihcr 
employe on ih€ individual's behaif Changes in ihz Uvel oi 
U^,^fcr pa/manl^ -hich rr-uW "r^^- ^'^^ y'^^^'^'^ - ' ' ^ 

included in caJculaiioni of the effccn on pn^iuaion und 
c^pabiUiy since by deflniuon -n-y are :-n^mfn!A for ^^h-- n^- 
i.erv!ce has b«n ptfiotmcd. 

ExominaUon of the effecis on uicome dbiruiuuan^^ind ty 
welfaie cliwccie wili require different ciilculauom In dt^imv^nn 
y| aggf^pie pfo4ucUon, it ;Mdc no Uincrfn^c wha^^^ininH'^ 
Mfcre incr^«d* However, when we coasider uicome distfibution 
and mrm^n% the e^ffiing abiliiy of ihh poor, aie Inierated tn 
ihe spedfic increments for ihh group. For txMtnpk, a nhort coune 
in wgiUni room, manaiemenc mi jh? aJlow dociors to ^ more 
patienff per day and increase the nhysiaans' incomei ThB 
increases ag^egnie production noiiibiliues but affecii ihe 
eamiosi distributiM by ikewing it even mor g to f he nght and 
!iiUe^lo:iake IndiMdyal^ off of the ^^dfme Thttttote, the 
amount of tncrewe in ear ninp for ^pedfic target ^oum must be 
mwui^ when i^he di^mbuiiofial and welfare irnpaai or wm 
progfams are considered. In addition.- whereas the sociai 
calculaiion of .ihc producuon capsbilny impact did noi intlyde 
iranif^r payment, transfcf' rv^'^^ ^nchid^^l ?n Irv^vinR a? the 
dismbution of income- Likswi^?. concern ^^hould b<: ^^^nh t^ie 
income distribution nim i^^^' ^nd no! before ■ 

CHher lypc^ of economu: nnrMC?;. ^ifc ihc effect.- on govcTnmcn^ 
receipif and e^penditufei. The pfop^^n^^ thctrmhcs requiic 
governjnent ^penditurts. On tht oih^r han^j, to the c^^ent thai 
tbev ineff35e ?he earnis^^ of ih? mnicwmu, iht ptommi will 
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^rvic« *hich le*d to higbcf eaminp. fuiun go^mmmml 
^p€k^lumofllh«r beht]r .Hi tie.,^4u«4 " H«i«. fhetr can 

m^Lwh Jon. Vic^i ihe go^'en..^'- ^conorn^ ^^r?,! 

and th- inavWMK TTiew, ho*fvef . a^e md.i^a rof . ^ 

theoTTOt that OTploymcnt.fclaied ptoramj !e.d to^^;-.- r.^^n. 
.hev m»y ma*< a mote prod.ctW. i.b^' fo-ct . - ; - 

Pfrve-Hcm ami Uw enfof^t af«««. ^'-'^^ '^^^^J 
betier ftffliJy tluouih ^r^mM6 mn.r^^ o, L.a« 

*^^. .S«n to be be^<^ ««P* 



MB took m the irnnMrn brnma tlw fti^w^' uw 
^O^M^ «4ttti. TOi is, of cotff«. a Soof.rafli* imm^ ot 

Tke Cii^ of InpKtt 

mawrured tf p<^te. »«y be one oe budgti^iy WMUalcw. 
tWww, wh^ Itoli ite ntifls^ of tojafit* *W«h <m m 

Md 'twOT of Ac t«pi«*: wd 2) ibe ms£ mi m%\ ol 




Tim tm^mm of prtoriiiM to sfl«ciffc mpmM mmi W-sA* 
ulttflSMC'^poariWlty of ite-ds^oo rn^rn m whom. ^« 
evilu*a«tt iriU ^ Wivtttdj ihiiTptswft ibo»^ dedde Ois mrtitivf 
IntM^^^ of ^ mmm teipiwi . Ost «fl sip** hmmm, 

tm mm te^fiw of mpiommi'mm pimi^ » 

mvMm m m ODploymwi w»d ««iop of itoe pwii^paBW 

io^iiB tii« ib^ i« bett saiididaie fof mmlmmn u »«« 
iojKWJ fflust N ijpo*^ Pot taiaflce. for cwiaw piopa»» 
ttopotiiaa of ^«ikipofli.« *ho will haw N«n eonvlawl of • 
crtet Of even amtt^ nuy be«,»fiiMi My of «^» 

wosnitt mutt be nesU^bt* in thiA sifu*«ott tbe orotAswo of Hie 
amis^fflJ b^i^m topsfl would i«m juiiified if Owf? w«e 
se¥«« rei0Uf« Itoiwilons &ni wlfwiion of aJT«« daia woe 
cofily. On oiliff hiftd. If We pfwwn commtd of pio%teng 
eroBloym««-relai«d »«^« ».«rfefid«i, th« topstt could be 
, sufrwaniJ*! owiuinjf Mi impart HcyJtJ te« a roajor citof. 
m rase aad HHt of m«asyf«nwt mmi aij/- *« wmWwc^. H 
%hmA6 be ob^oiu itoai prebtefm wUi «i« in accuJtiiEiy weasuiint 

cofflffliiiifl* ortmcs of oito^f a*o^.' whsviof tt«f f ailw 
, otmsive CMd Wily) iwaviwnf Tfts m^i wHcciion t«» may 
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omwtifh ih^- valuf of iht iniotrmmn Yeu #a illu^iiafad, ihcr^ 
. Btt mm mmm which w relBijvcly msy m mmsm^ There ■ 
mhm ^h«, with a liiUe mou^f, im devalopmeo* of iifw ajfjd^ 

mp^nm'^t ^pa^mxomi m^mmm could ^ acc-ampli^hew 
VnfonmMtiy, howevo^, c^p^it nc^ hm W p<5!«Ufti mmm to 
ifnor^d iw often, paniaiiaily far nonOT^omu: ^mpacu. In 
mm\mm. n b our apimon Uisi ail ^tenujl tmpii^ts shouid be- 
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Com^ im ^vm iM Pr«id«at autboriauon to rwrgiaue the 
«^^rw^ of the federml govemme^t. One of the ifeas 

Sy^«tatS «^«« « unemployed, underwploy^. 

^efluBrovtdtei such sem^ «« the followjag: CETA, the 
uTSo^ Setvi«, WIN. wd the Une^ploymem 

from the Vctmns AdmMitration; Vo«aonaJ Rebabilnauon. 

^tmeoi of Health, ldu«tion. and Welfare; ^^^^^ 
to Use Dmmam of Commer«: vocational mmng 
proSd^ by the De^nteoi of ^fense; programs for tmmng 
^^loym«t^"^nd«s through ihc Uw En^r««^ 
"ssiitana AdmWstr«on in the Dcparttnem of f^«^d 
ufflifls 6( Indiafl^ by the Bureau of Indiaji Affairs m the Interior 
IJ^rpwtroeot. 

You, as an anMyii m the Office of Manageni«»i aod Bu^ arc 
aaked to to the impacts of at leait four of th«e pro^aros. In 
making up the Usi you ihould aitetnirt to dcmonsirMC ihe ds,,. .. 



^^"Sovid« ditaS'l^iwnrTr m^^^^ i3w mipacts listed m 



ChapteriS 
EVALUATION DESIGIN 



yrtian evaluating social proyamst we ire ir^g to determine 
tim ^ttK^ of ttese prop^ams on the individuils who pardcipata^ 
the gjpvmjtaeat, sodety* and v^ous other parties. To do tMp we 
WMt to mwure change that the propmm has crMted—lJoth 
economic and sod^ change, primarily in the lives ^bf the 
partidpaiits. As discuss^ in the last chapter^ we also change 
ill tUe /^nomie ^litiou of the individuals to mraaure die 
t^Y^en^ of the pwffsm on sodety Mid the gdvermn^t* In order 
. to determiTC the ehug^ which have o^urred wittL r^pa^ to an 
indU viduai we owl to know what his experiea^ and lituadon have 
been after the trratment and what it woidd have been had there 
b^n bo sodal program, , ' 

AJterua#e l^^u V ^ 

. This diseu^ion ^ters around various designs which may be 
used tons^u^ the difference tetween the actual experien^ of an 
toiUvidual on^ he has complete an emplojnment^rclated pro^mm 
and hii apect^ experieim in ^e absent of the propmm.* 



sf mp^^mM dtd^t. Wt ilfoa^ urft ih^ tmdm^ id ©tamlse iome of the thwrpimd 
Btmwt i^d^ ©f^rtoteliU d^p. We ptnkyJ&rty rc^mm^d CinipWl and 



The Case Study DesigFh Probably tlfc simplest design is the case 
study/ Here, a tteatoent is. introd^ and the participanti are 
obsCTV# after r«»ii^ the treatment.; Then, baied on a guess as 
to what wouia haWhappen^ to the participants had there been 
no' pro^m, a judpaent Is made as to whether or not the 
trea^ent improve the lot of the partleiprats. TWs kind of design 
is quite ttjmmon i» the evaluation of social propams. Many of the 
manpoWte propams befuri in the ISfiOs gtm out of experimental 
proffams which had no evaluation other thm a case study. We 
have subs^uenUy found Uiat in mmy initwces the expanded 
■ progttm tos been relatively unsuccessful: 

There is a variety of m^or problems with the internal validity 
|the accuracy of the findings for the group. studed),of the case 
study ^^Ign which caused this unfortunate result. First, there are, 
events'outeide tihf control of the evaluator which influence the 
observaUons that he . makes.' To give a concrete wtample, 
employment-reht^ propam piacemenis depend for their succws 
largely on the labor market for ^^Uch they provide participants. 
During tte mddle 1960s the labdr market improved tpnsiderably, 
and it appear^ that any propam which provided warm bodies 
could find at i«st some job for these bodies. The improving Jabor. 
market would lead to IndividuJs gettihg better jobs ^an one 
might bthenvise suppose based on the experienc* of the early 60si 
When tte economy slowed down in the early 19705^ the op^site 
situation occurred. Thus, there is a very peat opportunity for the 
post hoc ergo propter hoc mm io^T^st. The evaluator impUcitjy 
. says "In my opinion, the participant was better after the 
treatment; therefore, the treatment causeH the participant to get 
"better." ' . . , ' 

Second, ttie mere passage of time will influenBe tlie expected 
labor inarket experience and the attitudes of most peopIe.t^This is 
particulari^ true for the very younff. For instance; teenagers have 
extremely high unemployment rates which tend to go downjas they 
• become older and more acceptable to employers. Consequently, a 



2. CampbeU Mtd Stanliy (13) cali this histoTy. 

3. Ciinpbell and Stanley (12) call this maturiiiori. 
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^ ^ takes in a youth whm hi is 18 md turns him out 

/ vrtitn he is 21 Us emplo^aent possibilities havep 

J f improy^ att^ ccMopl^taf the prop Some propuil, such as 
the Nei^bdrhobd Youth Corps, have been characterized as, 
. merely v'aipng vats" for ^A^^ ' 

Anotii^ problem with the case study is thw it is/^fflcult to tell 
if there is a selMtion problem, particulwrly in the case of 
. , sm^-scale danonstration projects, "nere is the distinct possibility 
that a sm^-sMle project wU be able to "cream**=take only the 
butter proip^ts for employment success. It is not surprising, tiien, 
that after having receive the^ft^eatment of the propam, these 
small4i^e projert participMts are found to do better than the . 
average person. . 

B^fort-and-After Deigns. While the case study approach has 
.bten used to make many actud policy decisions, employment- 
relate prppim valuation h^ tended to follow a sli^tly more 
sophisticate desi^. Tie ^fore-and^afte^ design has been used in 
a nuinW of jov^^nent publications describing tiie beneflts of 
social prop&ms. The selection problem is removed since a ^ 
compwison is being made with Ac same individuals, i.er, if they 
were above average in employabiUty after the program, they were 
presumably above average before it. The other problems still 
remmn, however. In fact, they become more serious for we itfe 
/now consider^g a longer period of time in which events and^ 
maturation take place. 

Probably the most serious problenl in using observations ^fore 
and after proff am parficipatidn for the ev^uation of social 
p^dgi^l is the problem of ^^regiression toward the mean." Itis an 
observed phenomenon that in large populations which we 
examined over time, those values at the extremes tend to move 

' towstf0 the middlei To pve a speciflc Example, if we look at a 
cr6ss action of hours worked by the labor force, we find at one 

^ extreme Individuals who are employed for a fantastic 5,200 hours 
a yew. At Ae other extreme are Individuals who are unemployed 
for the entire year. If we make a crossi|ectional anriysis of hours 
mployed in the following year we find the sjtme situation. Some 
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inividuals ttf worUui more than .fuU time , all the time, some 
individuals are totay unemployed. If we foUow the individuals in 
eithw of ^ese Mteioriw, ftom one year to the next, however, we 
wili finithe individu^ who are working the most hours m ttie 
first ywr have a greater tendency to reduce their hours of wdrk 
than does the population as a whole, so that their average number 
of hours worked wiD faU relative to the popi^Uon. This is not 
unrMsonable, for th«r have more hours to reduce. At the other - 
extreme, the individuals who axe totally unemployed m the first 
year wm tend to havi greater than average increases In 
emplo^ent during the second year. Again, the only way they can 
move is up. , % - , ^ ■ 

The problem In terms of the evaluation of employment-related , 
programs, then, is this: the mMual who is eligible'to enter these 
orbgrams is typically at the bottom extreme of the labor force 
specmim. Most of the programs require that the iridividu^ be 
poor, unemployed, underemployed, on welfare, handicapp^ or 
otherwise in need in order to qu^fy^for program assistance. These 
types of individuals are the ones who would be exp^ed 
subsequently to have higher than average increases m their 
. earnings and empioymeiil because of regression toward the mean. 
Thus when they do show increases in these vanables, it is 
difficult, if not impossible, to say what amount of the mcreases is 
due to the program and what amount is due to regression, toward 
the m^an. 

We consequently, have three major problems with simple 
before-and-after evaluations', the influence of extraneous evente, 
/the mere passage of tune, and regfesslon toward the mean. An 
alternative formulation of before-and-afte^ studiei attempts to get 
around some of these problems. This method, labeled toe 
interrupted time series, makes iwpeated observations before the 
proram and then repeated observations after the program. From 
the rmeated observations, some indication of the |eneral trend 
caused by maturation should become eyiderit. This method should 
also show some of the regression toward the mean by evemng out 

- — : 'vi ^t.- i„. .•„«„«.nf nFnhienis M Well. Sm CiinobeU and Stanliy (12). 



the; cj^e t^dugh whi^ individud is going if a numbar of- 
obi^atiou ov^ a considfra^^ Uma.can be madfr 

Hdwev^i such repMted obsi^ationi are not possible or useful in 
tbtf ^^isii of new en^ants to the labor forw who have no past 
apertrace to judp and r^trai^ labor fbrce who have 

been out of Ae labor force ^ - Idng periods of time* Moreover, 
tUs procure may r^uire continuing observations of the 
in^^duali'Over a long period' before they are allowed to go into 
the progruit which i£ obviously an expensive as well as 
time^onsunu^ proposition and as sudh probably not practice in^ 
many instant. : 

; Another^ altmative is to try to; predict the expwted 
befor^to*After chuge thrpugh the use of multiple regression 
analysis. This procedure uses such independent variably as age to 
cov^ maturation, anci powth in the ^onomy to try to account for , 
the problem of intemning events. The assumption is then made 
that the predicted mrnlngs of the individual resulting froi^he 
propam are net of these influences^ This methodi however, will 
not handle the recession toward the mean problem. Furthermore, 
it is very difficult to ar^ve at a regression n^odel which accurately 
sp^ifles the relationships of such variables as earnings and 
employment witti explanatory variables. Studies using cross* 
sectional data which explain 20-30 percent of the variation in \ 
• earnings w emplosTOent are considered to be quite good. This 
leaves at least 70 percent of the variation unaxplained, 

" Comparison Group Designs j To get.around^a number of the 
problems' involved jri the before-and-after comparisons, evalua- 
tions of social programs increasingly have attempted to use a 
comparison group to represent the expected experience of the 
participants in the propam Jn the absence of their participation. 
The key to the usfc of comparison groups lies in how well they* 

■ • I J, . f ' . 

S. Wf faM aa '*£ompar&on poup" any gtoup Whosa a^pact^ ahafietaristi^ ftttd labor 
markat oulQoin^ might not ba identic to those of tha pariidpanis in tha.ibsanca of 
prograin parUdpiitfon, A poup whieh was fandomly saluted from tha iame poputaUdn ai 
tha ^rtidpants but not'allow^ to participata wa labal a control group sincai if tha 
fiUfl^bara are larii' and the salaction ii truly random, thay should ba idantleal to tha 
partidpanti. 
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repr^t tto ejipc^encf that the propam participMts would have 
MdTthe Si^ee of the propam. If they do no «lo^y 
approrimatc the expect^ behavior and expenence, aU of the 
problems previoiuly discussed wAe into pWy. The ^'f ^ 
See Ais isto view ^ propam p^dpants as blwlryouth and toe 
^mpwlson group as wUte p^W males, and then IwM . 
^q^tabor market 

^p^tage r^urtion to «n~Ptoyn^^*«^*^Ki^^| % 
U conrttoaWy peatlr in an economic, upswing and low of 
employment consid^aWy p^ter in an «onomic dovms^. 
Diendfcig upon wteh stage of the business cycle we ejamta^ we 
woSd get different reidti from the program "P^elira* t^eri^ J^t of 
the choice of the comparison poup. SimUwly. we know that black 
youA would gain considerably more to emplopent fropi toe 
passage of time. Therefore, particularly with a ^O'^-^d-f w 
Srisonraia-proram poup (the black youth) wo^d show 
B^ta- gains than would the comparison poup of white Pjtae age 
El^dless of what th| propam tfd for the individu^s 
invoiv^/ Hi*, te te«« regrwslon towa^ ttie ^ 
youthi vrtio participate to th^ are pr^ably among the 

most disadvMt^ in society. They have nowhereto gobut 
wStoe w^ prime age md^ probably are at their p^s^d 
daunts tts tiiey pow older. The difference ^betwm 
the treatinent group and the comparispn groups are vital. 

Several sourcii have been used W semifp mmikm%^ 
comoarison »oum. First, evaluations of on-going progrMis have 
SSSS^o ha| been appU«„ta to the referral^ 
(e.g.. tiie Employmeht ^ct) at the Mme time as 
oc^ed but who did not go into the program. This is^ not 
nec«sarily a good comparison group because the« tn^ividu^s 
could differ from the program participants in the foUoyang 
respwts: 1) they m not go into the program because the^ had 
o?to of emDioyment or possibilities of employment which they 
considered to be iftetter than they would have after completing the 
^o^; at they did not go into^he program because they did 

■ 'e TouMtoBpWl'i and Suwliy'i (12) terminolosy. are Intertaloni beiwem 
MfKilonwdhliory.miiuritlom 
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not At qualiflwUons ni^^»ry to compleif it; 

3) ti^ oiHuiW from pr^M bmiise tti^ did n« m^t 
t|ic entrance qualifl^aoM; and 4) Iwk^ the motivation to 
enter the progf*m*'Orie roidd argue in the ftit ope t^t the 
comparison ^up j^ sn^or to the propam partidpants ud in 
the other ttiree oi^ Aat th^ are inferior (in terms of eirpwted 
labor i&r^ outeomes). A pMt*pregram observaUon of both, 
grou^ Should definite «p^ ^ find some dfffertnc^ In thdt 
experlenees* attitudes* and , bchavioir bwause of thw four 
f^sbns^If 4 Wore-aad*a^ were used, it would be vriy 

cUffleuIt to say what tiitmetionMftfff^we^ bttw^n the poups 
ud hist6ry» maturadon and ^fttdon toward the mwn would 
@cpe6ted-^but th^ certaidy could exist b^use of the selwtfM 
pro^. . ^ 

A similar t)^ of situation e^dsts for Moth^ eonpon 
i^mp^dn poup-Kmi madt up of Indlidduals who a^pUtd to 
the pro^iun and who were deemed qualifl^ but who did not enter 
it* TTiis o^mparison froup prtsuroably Is more able thw that 
deioribed above wttgh inelud^ who did not meet the 

ehtrmte ^MlifiaUoM. But it stiU m^y not be wmpwable sln^ 
theproblems of self-^le^pn (Items 1 , 2, and 4 abbve) or propwn 
iel(btibn» if the program took the most able of the qudined, 
remafii. 

' Other sttJOdi^ have selwted as comparison groups people in the 
same ndghbOThood who were in the same conditjon as the 
partidprats before flie program. This group is ©nestep removed 
from the applicants in that they presumably did not apply for the 
propami perhaps because they lack^ the raotivaUon to do so/ 
i^iitf^^ tih^lar group used particularly in studio of voration^ 
juration and youth prbgmms are individuals whose nam^ are 
taken from the fU^ of the high schools attended by the 
^artldpants^^b^^ who do not enter the particular program; Apln, 
tihere are problems of motiration, TTie qu^Uons arise as to what 
factdit not on the file wd differentiate ih^e individuals from 
4hdi^ who en^^ the program and whether or not ihm factors 
would in turn i«d to difference In the outcomes wWch a« to be 
obser*'^. ^ 
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In tU of th«e ^ of MinpMiwn poui», one an »tt«»pi lo 

To U» mmt that the two- groups differ in idcaafiablc 
cto^iiri& whWi we thou^t to af r«a the ommmt m«sw« 
and wMch «i qiil»tiflrtl% the sdertion prowdwt r^^°f 
te^Q^ «ft be to take some account of these diffeienees. 
A proWem In that our rabdels of what «uses the outcomes 
are norweU wmplete and are m<«uf wth error w 

that the r^E^on analysw wai not- m«sure «U of me 
difference.* Con^uCTUy. the matchin« piocmi dom mi 
lutrantee that the ^dpants and comparison gfoup mcn&ws 
come from tte tame poptaaUon. One person may he at the top or 
a low dUstribuaon and Ihe other at the boiiom of an overlaying, 
Wgher-toibutlon. If this is the case, they wiU r w«s to dlf rereni 
means. 

Also, there U a more basic problctn in that we are unable to 
tnsifQre many of the variables which we believe affw the 
out^met of sodaJ propyl. For msiancc. motivaiion must be ao 
iffl^nant factor. *ul it is difricult to measure on a 
betoremnd^fter basis-and impossible on a reirospwiiye bam. 
For th«€ rwsons. although they are desirable, rustical control 
mtthods are unlikely to, solve the problems of ihe com{»rison 
l^oup dlfforeno^ meniioned above.* 

Random Control Gnups. The answer to these problems is4he 
. I, „. .„:,«^f,„t nf fUfWi' m-T'^n^ nartldpallon or to a 
comoi poup.- AU of the perions who are qualified to enter the 
propam must be coniacted immedimay before the start of me 
program to find out if they are siiU inietcfied m enienng The 
group iUU ^ihlng to be considefed would then be spUi rando^y 
with only one roup assigr«d to awually enter il'^P^f^'f^I^* 
second roup would be »ven ihe regular ^^^^[^ 
. avaUabie to them. It is possible ihat this meihod could yield too 
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to te^^ f« iMly^' pmip wmtof lo 

the p^^OEi ^OTld iffadf^ &>- cMs dytficw^tic. Tmn 
M^mb^ M(^3^^ pf ^wti0». waiiW be x^ri ffm «cfe of i^t 

of M^ g C^i^^iBy s^sd wbm ibfy a^e ippD^ to Urge pwpt 
tojiitL- prrtwfc^^ wfi be pw. to Aite^^ b tbe 

^waof tb|tw pwps.* Ttos viouid W4 be trui of oi^ wa-u 

Prmt^i Pwhkm of Random Amgnmmt. Sr^efii p?otekms 
art favoJvrt ix3 ^ fMtdraa lekctioo of a octroi paup. Bm 
is ftiwtwct of IM opeftuni a|5«^» to fu-iiy 
qtiiiirtrf p^tf3^ frcro rewvini. ih€ of il^ a|i^«. S'«to 

m b fdt by the proffta opifitcrfs to b€ iM^ra to t 

doctof tr«toi i ^titni ^ho hu s ii^c wto t drai u 
avikai^k. To ^ury tbt analogy furtte. bawr^^^, %t d^X454 m 
m^dw U^t dniis ^ t^ed btf o-rt tbfv ut aimtesw^ so tto 
th^ tiftm Mt tao^m. TOj toting i$ dMt thxcMigfc jwn tfe 

rij^^rty. ,Om poup reives un^ orui 4m ot«T po^up 

^ual exacts of ibe^dry|. In maiWftf . U u rtco^i^ tfctt i:: 
^ could be mwh more 4iMj.iJro^i to MmiiiiStet a^nij ^hirii bjj m 

elteos or w^cti b« ^^ou^ sid* tj' fects of mi tot«i4 

bdMite4jb€ drug Is btetfteiaJ ibM w im iht dmi pfo^iy iM. 

if HJi tetfterive, s^tetltute a fretttr aJtrTOtive. Th^ U£m 

»d abotit lodaJ pfopams. 

who go imo the pTopami whi^h m4> mttami itom tfca^ 



»-t-ct r.i-aaifi. ^u- '-^r* "^^^'•'^ 

'^-*!*' •^«cns*:5 *.trt " 
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■4 fjucpte of mennt mviafl Thv-^ wmow nsm mi he 
mmMmi from om locatjon 10 ihc nm. ConcavabSy, we can 

■chArtoaiic ditfctor. bu» ii may- net be pm'Mt w tKtmia hjs 
thMisaa in Imdm m a numiMf of other cum. Com«?u«f]y, tne 

001 fixa®ia« site wmponeats of mt ptopam ajiiJ the tmmi tor 
jt* jiictta* andiaiJiMt. The pjopisiw is tieaiso ;i "5?i»cK hor- 

Tbe« prublems «a be todlsd in a vaiie 1 y of ^^ay s The pf««J! 
iotMfitlDi wtib the W«aim«'T is pmbabiv the easifl*' to hojtdlc. 
One n«4 eiay exclude iii< pmmt. Tm mc of ihc "post mi oniy 
dte*i«B" a«y aiw b« ^ removing ihc Ha'*tb«T3e efffict for 
Uwpan^c "Mis. sirm lAev noi Mrtow they are b*ln« ev«Ju*i<4= 
in itmi of ite pf«0W s«f J. one mn wy not 10 ooj^^ Jf« 

do AiiCTfutively, ooe eaa cvaJuaw s sufncifflUv fajge groMP o! 

■ mm s eoeoAb twitod w ifeAi only a small proportion 

of any ©voi mwnm is to be Miaplwl, TWs maJcw u extrereeiy 
diffl«iirfof the proiiaffi ffiaff io dedds who gm spmM 
ummmt, ¥m tiie probto of wif-wlcttioD and wramini. om 
«o aiieojpi io divenify l*s wcs. o! ptoiram paniri^^nw to be 
studied " One can rcpetiedJy tvaJuaie so iJai progrtm is 
studi^ under tn«y affcrsfsi mnmom to swouw for i^e 
ptoh^m of Rsat-raiians be%-0Rd «f ettrRoraw condJuoEi. 

HMJly. if iBete is a "black bo«" pfOTiem. owt can aii^m w 

. flb!«f^c ihe optfauon 01 pm&nm of w Qimmmmt mc 
piopom inw iwtampoMnb mm then tfvaioais the ii^wivcnett 
of men of ihm^, 

Aiiofhe: fmM^ pt^.ulm^ w^ih fundom immnffltm 
mm becau« Knawlfltliort wmi ptonnm ar. TOfi;:avab y 

'ymtm are iefum md wt wd, vidua] is wid mx m is unaualific^ 



for ihe program, this may be one motc^f a set of disappomtmenis 
to him which reduces his desire, moUvaiion, and abiHiy lo 
funciion. If thy is the cmc. then the ^elecuon proc^s*^ uself must 
b€ coMidtred as externally invaJidaiing the resulis by giving a 
j>osiuve bias to thg measured success of the program. Therefore, 
one must devise some kind of assignment process that in neutr^ ' - 
115 effwt on those who are lo serve as the control gt 
Unfortunately, it is very difncult to design a placebo sociaJ 
progrsnu Sii>^ social prORrtms are widely touted in the press and 
in various ouu«cn functions, it is difficult to tell ariy individuai 
that he did not rwlly want to enter the program because we are nd! 
suic it would be good for him. Whereas some sociai programs may 
not do ytry much for m individuaii it is hard to design one which 
w^ili have no positive or nef ative ef f^ts and yet not be viiwed as a 
sham by the psjticipanis^ This problem h a major issue which is as 
>^ unresolved in progiam evaJuaiion. The Mswer probably lies in 
honesty. The t>otemid clients should be lold ihat some will not be 
emoUed and. that the decision will be msidc randd^ly, 

FiftaJiyt we come lo liic problem of a lack of independence 
between the treatment group and the control group. A program 
may be dwign^ to Increase employment of panicipants in aj,maU 
labor market area where there is only a hmiied number of job 
o^niiigs. In the absence of the program, the jobs would be 
dJitribut^ randomly among the unOTployed, whereas when the 
propam does exist sJl jab openings are filled by the program 
participanij and none go lo the contro! group. In this ^iiuaiion the 
control group is not a pood proxy for the experience of the 
program participants ui !he absence of the program. As; 
postulated J in the absfnce of the proEram ihere would be random 
distribution of the jobs, and some of the people who hubs^uently 
panidpated in the program would have been hired. With the 
prograj!!, however, no one in the control group will be hired. In 
this case there would be an ovef^tatement of the incfementai 
benefits of the program, 

Thr !5ck of iT^dependeticf? mnv work in uic oppoiiie 
direcuori if the benefits of prngfam panidpation are transferred 
from the participanis to the control group Such a sityation is 
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«,n« Ukelv to occur when there U close contact between the 
post UMiy w uw^^ui c Thi^ mav occur wh§n 

o'ion *hich, is «sy .0 convey sue as ^ow 'o «r .0 

wouW Sv. happened, in .he absence of any pmg«am, .f h 

' if there wera no propam. 

nia^Sental and control groups, and should attempt 4o 
^^^f ^^dnanti from, becoming such a large faction of 
SrSt^ gfSSat^hey overshadow the control group. ^ 
Alternatives to Random AssignmBnt. In the situation where . 

S^'u iVsunley (12) cffer an ^^«''f'''-^l'Sr^ 

selwting individuals for adnussion.'* For ^nst*""' ,f *° „ ^.g^g 
sei^uim " _^ mav^be admitted and all those apove 

with incomes below S3,0UU may m aum aotitude 

postprogram QUlcomes are ic^i^- 

cvmlusiion. 



51 



rankings of the two groups (e.g., incomes or test scores)* If 
sigmflcant differencas between the two groups appear at the 
cut-off point ($3,000 or a grade of 50), this would indicate that the 
program had an Impact , at least at that point. 

There are several obvious virtues to this procedure. It allows the 
program operatbrs freedom to make all program assignmentSi it , 
does not irivqlve any unusual effort on their parts and it can be 
adapted to many types of programs. However ^ it also has some 
shortcomings. Firsts there should be no natural discontinuities in 
the outcome measures* Second, the effect of the program is only 
measured at one ppint=the technique does not allow a 
determination of whether the program would be equally effective 
for persons with incomes of $10,M0 or test scores of 25, Third, if 
the relationship between the selection variable and the dependent 
variable is not properly defined (e*gM a linear regression is used 
when the data are not linear), incorrect estimates may occur. 
Finally the technique does not solve any of the external validity 
problems discussed earlier. 

Another alternative is often proposed if the evaluation seeks 
only to determine which of several yrograms is preferable. Persons 
qualified and interested in participating would be randpiniy 
assigned to one of the programs. Recruitment would not exceed 
the number of program slots and a control group that does not 
enter aliy of the programs would not be necessary. Persons in each 
program would serve as control group members for comparison 
with the participants in other programs. 

Notwithstanding the advantages of tins type of analysis, there is 
a problem in that it gives the increments in the benaflts qf one 
program over another au opposed to the increment in the cost* 
This does not yteiii \/hat it is feally necessary ic 1 now* some 
measure of the ratios of the total benefits to the total cc^ts for the 



15. Thv vmdti U s^fongly urgeii to read Campbill (9) and Boruch and Riecken <4), pp. 
8?-l 16 far more dbmisiion of thr L^grasiion-diseontinuity dti^in. For initance, it might bf 
argii^ thai if the rehiioa'jhips bod. sides of the discontinuiiy arg Properly deseribed and 
id^fliical, then one mn estimate ihe impact of the program at points other than the 
discontinuity. 
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two programs. For instance, we may find that Program A ©ay 
yield a present value of S500 more in lifetime earnmgs for an 
additional cost of $300 when compared to Program which 
'argues for investment in Program A rather than Program^ If, 
however, Program B is providing a present v.alue of S2,0(^ in . 
Ufetime earnings at a cost of $100 (as compared to no program) 
and Program A is providing $2,500 worth of Ufetime return for a 
$400 investment, we can get a return that is on the average more 
than three times greater from Program B ahd obviously should 
invest in it. Only in the situation where we know the cost and 
benefits of one program as oppoied to no program or in that 
situation where the two programs have highly comparable costs 
can accurate interprbgram comparisons be made. 

The Choice of an Experimental Group 

Of the persons assigned to the program, some will drop out 
before they eyer enter thf program, others wiU enter the propam 
but wlU lekve before it is completed, and finally, there will be a 
group" of completers. Among the control or comparison group, if 
there Is one, no one wiU participate in the program. Some studies 
have argued that only those individuals who completed the 
program or only those people who completed the program and 
made use of it should be iMjuded in the calculation of the 
program's benefits. Such a proldure must make two assumptions 
to be correct: 1) the individuals who did not go into the progrun 
and the Individuals who dropped out of the program wfcre totally 
unaffected by it. and 2) these individuals did not differ m thwr 
expected postprogram experience from the participants who 
"completed the program. Such assumptions probably are not' 
warranted. 

It is quite possible that those individuals who dropped out of the 
program gained some knowledge or work experience whUe they 
participated which might subsequently be of use m the labor 
market. On the other hand, whUe in the program they may have 
lost time searching for a job and may, have misled job 
opportuniUes or they may have lost seniority if they had foregone 



a job in order to tmer the program. There may also be a stigma 
attached to thMa in ttiat employers might consider the dropouts 
unstable because they did not complete the program and not want 
to hire them. The individuals who did not enter the program or 
who dropped out pf it also may feci rejection or lack of ability 
which could subsequently affect their functioning in the labor 
market as well as in their lives. Similariy, the program may aff^t 
all the people who complete it, not only those who make use of it. 
Betiefldal or negative change in attitudes may occur among 
program participants even if they do not appear to be affected in 
their employment or sotne similar outcome. If any otthese effects 
on nonentrants* dropouts, or nonusers occurred^ the effects of the 
program on all participants must be. examined if a full accounting 
is to ^achieved.** . 

With regard to the second assumption^ the control group is used 
to repr^ent the aggregate experience of all pcuuns who were 
selected for the program* Unless the participants are a 
homogeneous group, which earlier we argued was unlikely^ it will 
be necessary to separate the control group into segments 
conesponding to completersp dropouts, and '*no-sh0ws- ■ in order 
to make comparisons. Such a division will be Very difficult since 
identifiation of the factors which led individual particlpMiti to 
complete, drop out, or not enter will be required. As discussed 
previously* our ability to measure and model such factors is \ery 
limited. For instance, some studies of employment-related 
propams show higher postprogram earnings for persons who 
remained in the program longer. There is no good test to 
determine whether this was due to their learning more from longer 
participation or whether greater molivation caused them both to 
remain longer and to have higher earnings. 



16. Tliis dc^ not' mean that no disUnctions should be made batw^n different ly^ of 
propM partidpanu. The analysis ihoulii setk to determine ihe difference which exiit 
betw^ eomplctert and ^o^uu in order to detennine the nec^sity of ,f fusing the 
proportion of dropouU or the possibility of jhoitenlng courses, 



relaUng to the proy*™ s oP^f""^"' "^^^^^ impacts and costs of 
^hlchmoveto^^rty^--^^^^^:. L.ious f.r pilot 
the program. Policy n^^''; : ^ mpmdtd to nai.onw,a« 

programs Which appear ^'^^%'?^^^,^,o{tM^B^n 
status, often do not want to ^^'^ unow which of the 

Soriprc^ding.Pro^,;--^^:^'^^^^^^^ that they 
aUernative fundmg Pp»£ p^grai^operators wa«t r«ults 
can make ^heir annual ^^^"f^^^/ ^^ ^^ and wiU 

---- 
I) «ubUsh a smm of f P^^Z^„, who cannot perform; ^ 
train staff, .^^»nt out lhosaswrfmc^ _^ ^ ^ 

3) announce .i« P^^^-^^^^,,,dly. th^ steps ^vnU 
4^ operationalizf the prograni. efncicntly- If the 

S^time ^tore -^^^l^rewUiprob^^^ 
prorwn«e^ *lwted before has .^p^^ „ j„ 

. LMownw«dbias,nt^«uin»«^^«^P^^ ^ 

addiUon. it ^^"^JZn^ltf^^ of the program to be 
program treatment for i^^^^^^j ^e removed ftom the 

dissipat«l. Bnally, »*^"|,^;^,^ass after treatment before 
data by allowing at least a full year o P 

I mnact measurement, . . 

impat.1 n.** evaluation of 

^?^^«««,ihi ending in Frtr^OT- 
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measufem^rus of program impaci should noi oc^ur uiiii! one y^ar 
gfter the parikipants have kU the program. EvflJuations should 
aJso ite mfldf at Ihr^ or five year iniervals afier iha program 
panicipaiits have let mJnated to find the icMcr run impacts. In all 
mm, care should be taken to insure ihat ih^dnra arc collected fot 
saeniical time periods lor both the c?^|K:runenuii tind i^untryl 
groups* Othemise* problems with cycHcal nuctuaUoits may arise. 
Variabies such as eanungi and employmeni should bi^,rrica.^urcd 
for the entire postpiugrara period as wgli as for the mdividuaJ 
years. Such measurements wUI demonsiraie iht lotal effeci of the 
program and changes in program effeci^ ov<^r utni* / 

Tlie Cholcf of Rttfwpeciivt * 
or Coacyrrtnl Bita CoUe^Uoo 

One can identify participant and companion group members 
associated with, ah ^rlier progrHrn and giiiher information 
reirospectJHely. Alternatively,, daia collcaion can occ«r by 
Identifying a group of future program partidpOTts and contro! or 
compariMn group members and then g^fhering information from 
or for them over succeeding periods. Time arc mmn$ to both 
procedures. 

Retrospeciive studies provide icmH^ faster; they do not retjuiie 
waiting for the posiprogram period to occur bt^fore measuremenis 
are made. This ihorier eiapsed time from im dei;i^ioii to conduci 
the evaluation to the preseniaiiori of ii^ findinp h the reason moMt 
evaluations have con3ucied in this nuuincr, aiihougl^ another 
argumeni in it^ favor h ilmi there will be no **Ha\%ihoritc cffeci** 
since the individuals and siaff do not know ihat daia wiill 
.>ubse<iuently gathered aboui ihern Furiher, tmo^nmyc daia 
gathering does not r^Uire that the cvaluaiion be built directly into 
the pfogram*s operaUon, which miikcA iidyanci? planning 
^ unnecessar>% . 

^..OPfteiting these facior^* concurfcru daia cojlccilon offm 
several impoitam benefits. Fiifst, it is impossible to have control 
groups in a retros^tive study becau^ random assignmeflt is 
Impossible after ftte program has ended It should also be noied 
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thai it wiU b* difricuU to impleTneni ihc rfp«'-ton di^mnumm 
design on a mroa-pertivrbMis m the wlcction aimvA wtil i)<,uaily 
ooi be matic wplidt in nornial progfam apcmmm, rhctciMfe. 
reuospcdive studies must caimma ad hr^; ,:Dmp[jr: -on poup^ 
rvom whatever records This wiony involvetj Mios(ax»«.il 
clifficuUy since tHOgrafflH mmm ir»»uiu»Mi M,u:u,Mtfi.v:. ,^ 
nonpartJctpantsMd w the Mtem that itwse li<,!s »ic irKomp.ctc c; 
inaccuraie, biaws may be iiuKMlucciL Nchi, tJiia ga'a'.crciJ i;;^ .> 
rwospcctive bMb wc much inoie hk^iy w mml^t f«po.« 
errors. The longer the period m m covered. Utc moie Ufcdy ^kj 
mwoory lapses. Abo, mdm sho. iliai awmasi; ^'^^'i^fJ^a^ 

bwauw perceptions of past auiiua« if-iiltcfcd by inicrvemng 
events' Finally, aiiu«*J earticf, rcuasr^uve swdira me mou 
mmio havc'inad«quatc f«|wn.w fate* H mo'^ dilliculs amj 
ioiOy to. try to \ocme indivtduab afifff mnw^x ?ui3 wen sffvcreu 

We find the set of wgumtniA foj mnmutm mmti lo trt much 
ihe-suon«r. In out opinioti, the apponmm to a conuul 
aroup Of the refiewton diMontinuity den^jpi oij4..M^i.r. us-. 

early, but inaccurate, infortnaiton Also, the ftftfuter and 
tower mn of wnw(«oi daui i0mmn tevc mwh appeaJ. 
ThetefofC. m sitongly wgc lU mc 

Coilf«rtlBf I>if« 00 Jrap«cw 

■ infonnaiion iteut the ptopm wniapams ajia ..^ntio! group 
meintwrs. Sp«a«i studite have usualiy p^twnai tincmtw,. 
while the govemmTfit folltw-up svmob Has tciiwi piimmly an 
mail QU»tiofWiMr« oiid telephone immic^s. The pfottlems w,m 
difwt comact w«» known t«-s,d« bmm '^^'^ mnMmmg »M 
-••ostly mp^nx rat** on p<ts6nai miemc^s ^Wom are -cjove w 
i<«cni'and ?ho« ou mmi mA tdcphonw survciA arc cfltodwabjy 
lower," Yd. 101 maiw vflftjibleK ^m^h m^i affected by s-ooa^ 



pfD^amK direct c^onran mav re ^ -?^.*> ■ - ^ 

Tti-e fM>ao^ M5rai0n fiethtug-^^ • r:: , 

the :iiaiV-idaJLil tor <mhom d4U ^^lU ^u^.v^^rr.- i-- rc , ij;/' 
kno^-^n from Ih-e ^tl^n of tH-* nr.-^ir/irn ■ ;:'^fC:.U - ' . .^^ ■ .^r. L"'-=r • 

Tiii change^' of addm-*. r -^rne^. im v.-; , . :■. -J . 

mntMCt nttdi^ lo be miifn^.neU n/^ r--^- ' ^ r ^k■: ^'^-^ 

c-^^-?it't Awi^l* r>f the r-?vfM\:.:^j< ic: Ui^': 

r^EiUiitioq of piog^arni^ :t?^> tt^^ 
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withto ttii« tB«ih$ of rte «id oft a auaiicf . Cofti^ti«u>% 
uiG^ptoyBifflit innmiMe daia do ooi wSim from wo of Uift 
irii^lw faidls of Mdal s^rtiy data. 

^ iMtirite ffliw of the inajor Mttstrtil 
Ivw to iio OTltott *e ^*»* coveiap to not mmM 

wM^-«itf^« paions wtoo work for iametfatc leljuivcs, 
m^^m of maJi «^ »oim« «D^oy«« of noopoOl uttits, 
Fqsarale ^ffla «&l f« Hw fmm. fedwml govwnmefli. 
wkI fiU»«rf pesoond. Pumhwmow, brauw ihc syMeiB Is 
.M«tj^l»^« is flo wuf to di*iJnguish iOdivMuato mm tovt 

or «lio tow teft lafcw foice. In oil tow c*ws ttec would 
m^mA oC otfWttp. Tto is ^j«^ly tapenajBi in Mm 
m^m^Ai^m^MM ^utOMim, mm m Ne* Voilt City aad 
pl^fflttari. Fiwilly. ilWsrtate usuatty tioW rajniDgitwofds for no 
more slwn abotttiwo y«i4. 

skort ttffm ffltow-yp in indmirM aim?. In c^m ^ 
provide i^iniite daift qitjckly for niOTi of the mmple. For lon^ 
periods when the p«s«ft Is more likely lo hflvc left the 'Male, sodal 
swurity daia «p|5W w is* a b-iiiei alieinflsive bcmnm Lhcy or* 

aifflpteSiis 

tte pfoWero of how n»any panidfWJsis Mtil wnttol w 
com^iMB yottp wembera should U examined Is alwayi a vey 
difflcoli one. Tl*c usual tiilc of thitrob Ia to include « wwy as Uie 
bud^ wiU «llow, Ai mm tmk wpWsuwied Jusliliwtlon* ptisl 
be proifidtti, fa tbeae iiiuaiiow the bfisl sttsteMf Is lo <ontact a 
samplliig«P«t, If one is notavaiIfflb!e, the siinpiesi cakulaUon is 
d^ndcni on the dispcfiioii of the variabic being siudlfd and rtie 
Itvei.of pf^oo dkW. The formula lor the standard cjTor of 
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limt, .^ top:' ^Mm^^^m mil fen. ^^c^^crf I ^ ^J>f^ di^pei^as 01 ttc 
^M%Mmm m sample i^} mmv^^ aray from SO pa^^fU/' . 

l^cfimm bur i^ho did cos mm k (probably ite piefoted 
po«P for fWffipc^i^e iiadto), m mmt idy 00 

of ite fffeTOl oi^cy. Tfce m of tuch i^cwd^ riefluim iht 
IdmMmim of iitt piott^s 10 ^ Kiydie^, ifct coftia^ttog of We 
local ciffira, a dmiW Um^^ of tteii rwm^.ro otiui? $ 
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ih^ r«erds of obabb ooi mnmn m infoiroauoo about 
SMsd lime WMumtoi. 

Pifv^^ in o»cf ^ 
family ffloflbws. to^ily wwrn^ 

. ^^^^ 

iBJiUflp 9^ work «peficn« qi«sisoo». -^&ch slJows lor 
ififegraiion of ihe wo ms 0! daw 

have MHkipaifid to the popam undei .to.y. the l^V*^^ 
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funimmmM pwpcm fm th^ CPS--io MU«t data ori labor fora 
beh«^*i«, Tbt Cwm Bu^Mu would hava to either to 

mdudf qumtoas on fia if^uliur CPS surveys or to conduct ipeciy 
mn^smft^^toe household left ihe CPS Uscwviewing cycle, Bmh 
nffwmmt w^pm% mMkdy ^ven the lefal and other cowtrtints 

The i^otid m^hlm tovolviK the qumJori of sample size. It 
might not be p^ibte to wmn a lar|c «nipaiiwn ffoup where 
mi^iAJiig k ^k&i on mwy cJuyracierisii«. h would tUo be 
difflcult to eoMtfwi larp maplm^io match pou^ who wmprisa 
m smaU ^Htion of the pepularion* TWs problem tiU£^ on noajor 
pf oportioiis wh^e matching on the bisfi of the individval*s work 
.mcp^Amm ss wed^ (fti It wiil be for most social programs Anm 
they (tel primarity whh the unemploy^) and where the CPS 
ffirk^ and OTploymeni data are ijso lo be used for d^p^dent 
vaiiab!^. Tlic work «perien« qu^iions su-e ask^ in Februaiy 
and April and ih^cojne qu^tions are ask^ in March* Data on 
tb^ vaHabte for the pr^^ing y«r (year* ' !), are availabje for 
appro3dnaateiy 50,000 households* If^ howeverv the depradent 
varkfck u data for lbs following year (year data can only be 
^cu^ for Uie 25,000 households who are in the CPS sample for 
tbe swond time (to ymr i ^ J). Furtbermore, subpoups may be 
difficult to find sin^ persons unemploy^ at some time during a 
year coaitituie only a pro^rUon of the non-institutional 
poptilation 16 yars old or ovo' (20 percffli in 197$, wUcb bad the 
Wghait poirtwar unemplojTOeni). Tbe existence of such problems 
ean be i ound b^ wmp^inj the si« of articular subpoups in the 
CPS with the mtaimum number of oom^ison youp memb^s 
r^ltured* Berame of its targe siw, however, the CPS should 
provide the nwwwy romfmson group in mosi cafes when fine 
br»kdowni w not n^»sary. 

Socto/ Set^rity Admin^traiion CWHS Data. The Social 
S^urity AdjnWstration malmains a special fUe c^l^ the 
Continuous Work History Sample (CWHS), providing a 1 per 
Mmple of individuaJs who have applied for sodiJ s«urity 
nureberi. Information availsble for ihm persons include age, 
race, wvwd wiungs, and employer industry and lontion 



for quart« wito covert employment. The usfr of these data 
to form comparison poups involve severe problems. First, the 
CWttS m» do not ttjntain identifiwtion of individuals, 
aimough ttoy permit linkage with tapes from the . J Security . 
Administt^ori throuih common case numbers. Some fonn o 
awmmodatton would have to be worked out with the Social 
Seci^ty Adintoistration if social security numbers are to be 
compared I|^^proffam participants to make sure that the 
ina&als io ^'^ntial comparison p-oup have not bwn 
piomw ^cipmts too. Also, since the CWHS contains only^a 
limits art of information, matching on these vanables may not be 
suffldem to selert a truly comparable group. Important vanables 
such as: education, marital status, he^th, existence of other forms 
of ttainlng, and famUy income are absfnt.- Moreover, it would be 
impo^ible to swire this inform., aon directly since the mdlvidual s 
addrete is not i part of the rword. Finally, the only dependent 
variables which could be examined using this source of a 
comparison group are the information on earnings, number of 
qiiarters of covered employment, and industry ^d Ideation of 
^ployef . While the information on eaminp i& a key measure of 
the succMS of employment-related programs, the, shoncomings of- 
tiie Sodal Security CWHS data would appear to limit severely, its 
usefulness as a source of comparison .groups. As was discussed, 
above however, social swurity records are the best source of 
wmiiis ^ for long term follow=ups of comparison gfoups 
sel^tal fro^«ther sources. 

National Longituainal Surveys. Four groups of five thouswad^ - 
indiwduals have been surveyed regularly smce 196^8 m | 
proffamsponsorW by the Employment and TA^ng Admmistri- 
tion of the U.S. Department of Labor. Tlie four>»tps are mal«^ 
bwween 14 and 24 when first interviewf d, female m the same age ^ 
i group, women initiaUy betwwn 30 and mA men 45 to 59 at the 
mst interview. The samples are nationaUy representative, and 
; there is'a tiiree to one oversampling of '^on^WteS' 
' cohort of 13,000 youth between the ages of 14 an4 21 be^gan in 
1979. This- ssmplf Includes ov« representation of biacKs, 
Hispanics, and the poor. . >, 
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The surveys includf a wide range of infoimatloii about each 
-^spondent, iajluding age, sex, raw, marital status, nuaiber of 
dependeBts, family income, education and traming, 'TOfk 
experienM, cMnings and income olt the individual pxid, spouse 
during the pr ecediag 12 months, current labor torce status, health, 
assets, family ^aclcgrbund» mobUity, and psychologtcal measures. 
In addition, these data are provided m n iongitudinal basis so that 
substantial prior tofdnnation on ihfi vjmpmhon group merabers 
would be avaUam FinaUy, iae d^ra are readUy accessible and 
extremely; weil dofcuinimtfl.1.-' 

There are fiso problems in using *hese data to select comparis »n 
groups. They cm oiUy be used for the sex and age groups that !hcy 
include. Even for these sroups there may not be enough sample 
members in small categories of the population. Next, if the 
dependent' variables are to be taken from the survey, the periods of 
work experience and earnings for the program participants must 
be the same as for the comparison' group. The surveys have not 
been conducted in each year. Furthermore, <he data 10= the 
lonptudinal surveys are coUecttd fn specific months and the 
program participants must also be Interviewed in these months to 
be strictly comparable. Also, the social Security numbers of the 
potential comparison grotfp members would hive to be checked 
against the national lists to remOve those persons who had 
participated in the programs. Finally, it is not clear whether these 
cohorts will continue beyond the early 19SOs. 

Due to the problems involved in the use of each of these sources 
of comparison group data, we recommend that they be used only 
when there is no alternative or as a supplement to other 
comparison groups. Limitations on the ability to match them to 
participants, in the methods of data coUection, and in the type of 
information they contain create many threats to internal validity 
and hiake their exclusive use hazardous, ' / 



2S.Dtwup«ra«y be purchiiri from the Center for Human .Resource Research, The 
Ohio State UniveTiiiy, at coji. 



Th/Cholce of Indipendim Variables for Anafpis 

To' conduct an evaluation of social programs, it is necessary to 
measure the relationships between the program impacts Jthe 
dependent variables) and a Variety of independent variablei, 
including the personal characteristics of participants, the program 
components, and the conditions under which the programs 
operate, . 

Personal Inde^ndmt Variables. These independent variables 
are particularly relevant for comparing program participants with 
the members of the control or comparison group to discover 
\hethef differerices exist between the groups which may affect the 
program impacts. It is necessary to include in the analysis as many 
variables as possible |Which are correlated with both program 
participation arid the dependent variables. However, most 
relevant dependent variables with which evaluations of employ- 
ment-related programs deal are functions of more than , one 
indipendent variable. To omit some variables in the analysis niay 
lead to distorted conclusions du6 to correlation or interaction 
among these variables and those independent variables which are 
included in. the analysis.". The analyses should treat all of the 
independent variables simultaneously. ^ 

The use of simple cross tabulations to isolate such relationships 
is inadequate in most cases. For instance, the effects of race, age, 
education, and skill level on earnings are all interrelated. Yet each 
of these effects should be distinguished. To cross tabulate by all of 
these variables would involve so many cells that the sample would 
have to be enormous. In addition, the tables would be so large as 
to be unmanageable. Therefore, multivariate techniques should be 
used in the evaluations to discover and test the statistical 
significance of any relationships which are observed. Multiple 
regression and correlation techniques can be performed with a 
much, smaller. sample than cross tabulations and permit easy 
interpretation of the findings. 



_____ ■ ■ * 

26, For a brief discussion of this pfo^em, lec Suits (13). 
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Evaluations should toumine the affects of the propMris on 
frou^ of partidpMtf for uther reasons tc^. The, analyse should 
dttenuine wbMher or not a particular program will benefit certain 
target p-oups for whom Uie proffrwas are deigned, as well as find 
whiph propwns stive the poups best. For most emplopnent- 
relat^ ^oprnmst Ind^ndent vWabte should be Include in the 
ai^yds /to reprint different pou^ with high percentages 
mnbng ttit poor and the imemploy^s such as blacks, 
M^ran-Am^WM, Indiana/ the handioapp^t t^nagers, the 
ag^, school dropouftt w^wnvietSi welfare jmd unemplojment 
.insurjwqe ri^piratSt and veterans. The dep^ of suc^s shoifld 
be mwsitf^ for such characteristic of the propira participants 
as age, sex, race, e^Aic poup, number of dependento, family siie, 
education, etc. Many of the relevant perional characteristic af e 
listed in Tatile 34/ 

It is also ne^sary to treat personal characteristics in the 
evaluation in order to improve the efficiency of the propams. 
Propams wiU'have varying results for different tj^ of people. 
The personal variables in Table 3-1 also fae u^d to determine 
which indiv ^uals get'the greatct benefits from each program, and 
individuals can be assign^ so th^t tho fucc^s of each program is 
maxlmlE^. The attiiudinal variables in Table 3-1 may be 
particuli^iy useful for th^e purposes* 

TUs assignment pro^ may be in conflict with the desire to 
benefit certain target poups, however, bwause these groups 
re^ve lower benefits than do other workers from aU of the 
propwns. In this case, knowledge of which prppams sew which 
grou^ best wUl still be useful because it still will be more efflcieirt 
to diocate the target poups 10 those programs where they, revive 
the largest beriefits. If there a^e stiU program slots, the individuals 
who would have the peatest expect^ benefits would be enrolled.. 




maMtal Status 



jfamliy eom^tilQD Iflcludiagi 



niimbtt of other famUy mfmbers \ 
agi uid sexof other fanjUy mem^rs \ 



otb^ famUy income and number of other f amilj^embers 
eduwtfonal level md 4uaUiy 
mUitaiy Msvice 
Itssets and debts 

empte^ent status at cnroUment 

ptevious oocu^tlons 
previous i^y levels and earnings 
rl^n left last job 
previous training 



sodbe^notnic 'background / 
job# and connecUons of frlen<h-and rclailves 

InleUigence level 
arr^t reMrd 

income maintenance staius at enroUntent 
eUgibiUiy for different typ« of income mainte 
tax rate on wrned Income 





Uc€Qses ud wrtJflcatlons 
knowl^ge of job seekuig skills 
knowledge of bow to keep jobs 



skills and abUitim 




mttiituA^ of mba mfflAbeis toward work 
i^^to teuaponatioii 



t<9 «prps oraUy and in wriung 
piidf io^obUily 

atdtuda toward wofk^ig 

gffiml disposition 
motivatf on to pi ^rad 

d^pree of IndfpmdM^ 
ievil of fnaturity of you^ 
atUtudi toiwd flrotptini wetfara 
iidiliQpitss to rdoMtt 

occu|M|iioiifil and pay aspLratK ns and e^^p^craUon^ 
percdvid limltaUons op abUtiy 
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Pmgrttm Componem tndtpmdtm Varkbtm. Mosi social 
p^mns Mosfafi of 1 M or activities, aad many of iJi«« are 
Mmmoo w acvefri propiTO. It would be cxtfOTelj. wftil to 
modifttog edittni pro*r«ffls ^ in ifte ptonuig of ne* prcparos 
to Icoow wWch of the eofflpeooats ate mm\ efftaive for various 
typ« of pinW^Dii. It woiild al» be daUtble lo have 
IttfomiUen 00 the b«l combtaatloiM of campoowift. To the 
exteot tbu the taa^b wbA wumt of ihe ffompooeow to 
iafS^Mll witWn or betwwa itw pfopMMt miJtivwiMe 
t^toiqUCT «ttt ^ uKd to Idtoiify efftttlve eow^o«ois. If 
eviaaattou gomiiie jwopw* which l^lu& « ^aiitty of. 
cQmpQii^ wrf wIteftAht kflnh of thf cOTpOMflW ih^ 
^ould iBdi^ « ifl^pCTdffll vtiiabto ite Moouni of ««n 
cervtee |>»fof^ to t propim (tW* wiU uiuaUy be «pf««>d to 
tstflfcoHiOuiii fprat pef puii«ip«nt) and, If poMJble, a mcaauit of 
quality. 

fiwgfftottf rndependtnt Varmies. EnjptoyTOcai.ielaied sodW 
^og^ms «Jm dlffw to thdr effectiven^ depCTdtos on 
chtnwt^ljto of tta lowdon wtd the drcuflMi«c« to which ihey 
op«t»t§. Amopi poientlii fa«o« tfrerttaf ptopMm itwoM ««t 
level of unOT^oymtoi, powth to emplbymwt, %smm ew^«h 
ami tbe iliifM of ffiinufaauriRs Ln ihf .^rea ^ whf^h the pofram 
OOTK*. Ac and nawre (firm, mml, depr»«d, ttcj ofpe 
a to wfi^the prti^m occur* and the.defiee of ^KfimtaAtfon 
Ihe urn miabt tJ^o be Ineluded. The type of skUl, the dOT«nd 
iQ% wotken wUh ht given iStiU. and iht »verate' eamlnii of 
peisoos with that skiU would be tapoiiant if iht ^opwn tovolvts 
tniWoi for ipedflc tklUj. Thtee variables shoidd be Included Id 
the analyrii m te^wndCTt vaiiabtw to determlae uodei what 
coodiitoni the propams are mosi effective and which pro|tajn 
ire most eff«lJve ufldeir parflcular conditions. 

OeiwmlBlBt Propff r^T«« Slie- M«««ftot M t^ M« 

' A bwic fluttUon *hl4li ihe cvaiuaUon sho^ild answer Is "WHai 
- ihotad be the ptOirsiTj'i iue?" (includtng ihe ponJbUiiy ihai the 
amwCT may Iw iha' m ptogram h jU'itiHsdi lJc«liy. ihe 



myiMl^fl wpiild prev^ m i^oununs of iM toiai N^nia 

Mom^ ^ mpk^^mm lo yidd ibc levd of wttviiy tm smii 
whW would tea^dmfa^ ! -Mr wil muro on iht lotal «p«diiiJiff . 
Jo do Ms, wouM lOl^w Wi ffswras w ihat «4h tddiuo?i^ 
doUar w« fppit os m^V^ ^Wch >1eldfd ihf pwiest ffium 
for thai drtiyr. tte dto^buiioft of pitvloia en^dimfi^i 

^ To dtte^ bowt%r: . anluaiJoQi of CTpio>w«^ftlaied proff mrs 
ntf ^i^ifM tfc^ daw. Raito* mmtsfi b«effti have toee^ 
cmteatoi^ for a ptotpmw ii fwt^ of pfoftim saiidiyt In 
to radtd pfO0«» ria d«e}^oas, uMfS of Uiei^ nm^iwm to^ 
Jiadi o^^imi^ ibai iht ivirtif bCTefiis of dif fcfral pt^m^M hMS§ 
'^dkmfdmi^MpiQ^tm Unfits nt sddif^ 
i praon to » ptw*^ ^t^^ ^ Wgh aver ate tetffli would N mwe 
bcnind^ etuoliing ibi peiwa ia a pioftAm a low^ 

dadlloii m^m^mA up Wiit ilit ^opUftif^J lUloc^riori of 

iheffftcij^ chfl^s« lit pfo^im mt. Of^t i«?wlmjqae ii lo ttmt 
the tteoluit ami ftteu^c atm m im piogtra in auHmfit mmt 
' markets \o ihf l^d of pwpsm itiiwv^ in thwe kbcr mtfkin^^ 
Hi^ii U a wde iMse of ptogr^xn ^h^ch mmy be asad lo 
pttdia tJh€ f ffeciA of ogi4m gjow^th or%yibaii.f . roie^^pie^ 

' itMuc* avaatf twiefits, ail oihe? fa^ ^is held €^i^iaiti , 

^ cKm^ occur in piopOT aii^. ihi^ ^^ually flitam ctofr^ m 
ihc ij-pe of pfopam paftia^:in^^ For tn^^i^nw, mall pfo^arm 
m^y •*ct«iii'' iftfl ontv v^yit hjgti ^tiool ^Mmim \Mttt 
proifims may have u? ai# d^pe.^ imd mroll dfofwyii. It is liktiy 
ihai piopamA to^^^ ^rftorail^ eff^i dc^odiog on fht typ^ of 
punkiiwinl*. Thcictor*, n thft cvaiaaiions mji ddcmiM the 
avctitf rffe^s of ihe p/^g/^m^ on ^ dirf««« gfoujM 



n 



" bt«ci€ Pp^TOiW the cf ©to profr^ 

14a ifaa be frtMferroi to oo«- Maa^b^, bomt^ tt, w^c 

the ^to^fii*, fM &mpem^ of ihe pr&piffl. ttrf ia ferefti 
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of daia on ouwomes. Tim should enhance the rc earcher's ^biUiy 
10 lomH mhltm-% for follo^-up purposes. B«aus4: of certain 
dm^backs assodaicd mil ach of ihmt oieth0ds, however, the 
bidd^ is ixpwed to di^^usi an ! rlffrnd the rauonaie for arid the 

aJfcrogUve. Opuonal apprmchP:, c^srefuHy considered and 
argtii^rf, may bt propOMd. 

The purpo^^ of ha^^ing a dcvclDprnen phw or woW^i^. (vi ^/ec 
VfhrlJief B mfihodotogy for meMuring ihe independent effects of 
counMliftS ^ developed. Thus ihe offlf^^ ^l^ttfd for 
cir^ynin^iUon hhouUl ^omc ^.c: ^!aj;daxd:, v.luch can br t^rcd 
!o define m ad^uate l^vel of counseling prograiB effort. If ii is 
no\ possible lo deviM a wmm of da^Ung differences in outcomes 
between lilt c^IwrOTentaJ and eoiHfOi groups wiihm ''e^erfsplary^' 
oifmt {Ke.. those meet ccctain siandMds used w define 
cminscling quaUty), ii would make liUle sens* to undertake a 
fulhscalc evaluauon ose sample would meiude dffk^rH which 
lail lo mmi Ittbs^ Im^ic ^mndE/ds. 

Tlic^t: a/c, ai the prc^enc um^. appro^^irrr^^^iy 2,4^) ES 
offim ihfoughoul iheeoynuy Of ihese. about 60) have staffs of 
20 OF more and employ at lease one full iime counielor, The 
iample of **exetnp!ary'' offlees \o be included in the research ^ 
should be drawn from a universe of approximately ISO offi^i 
^hich imvcMunseUng^cgia^s i\m me^r specified ^ttindard^ of 
adequacy. The DOL'will review the .tnpropriiienws of the actua] 
stii dards for sample Kl^iion propo&ed by iht bidder. 

The experimentaJ groups shouid be dra^n from iocid offlct 
appiicanis ^•needing" (sj defined in ihc ES^tH^nuai) and receiving 
coiin^eling lervice^. The mmpBximn groups should be drawn from 
lom\ offitt tippHcant? ^'needmg'* hyf -^^t receiving counseling 
semces. 

A$ a jummary checkiBi. p/CH^omls are expected. a,s a minimum, 
to dncuss the foljowing meihodoiogicai i:i^ues: 

Cn whether the reMrch will be c^irried out on a 
c^ncurreni or rcuospci:;vc ba^s: 



(2) procedures for .-Iccting the offices to be sampled 
and criteria U^r stv cting the applicant groups for 
inclusion in thr u; les; 

0) problems ouc ,..oiiU. expcci. to cncoumer in trying to 
c'labli'-h ••^.!M-i^biliiy hptwcen and among , the 
c%pimmnul and cDnu^Jl/comparison groups, and 
he mcthwi; thv be used to deal with such 
problems; and 

(4) the sourws of data iv . e collected, and the timing u£ 
data collection. 



Chaptei' 4 

THE COSTS 

OF SOCIAL PROGRAMS 



No program provides its benents free; something ^st be given 
up in order to derive them. Respurces which are dc^ uied to social 
propams cannot be used to produca other goods and services. 
Societyi by devoting human resources to conduct tfaining 
programs for painters^ loses the Sf^vices of the instructors as 
painters, as auto asiembkrs or even as college professors, to use a 
few examples of some of the alternative produciipn which may be 
forsaken. Thus, the costs of social programs should be cortsidered 
to be their opportunity costs— liie value of the altei native benefits 
which are foregone because of the programs. 

Esttmating Opportisnlt^^ Costs 

it h usuaUy impossible, however, to identify what benents are 
foregone and tQ place a value on them, especially for large social 
prpgranits. For instance, what production would be lost if society 
chose to require ^all students to have an additional year of high^ 
school? What potential output did society give up by providing 
medic^ services to the poor and old under Medicaid and 
Medicare? Cdnswiuently, because of the rtteasurement pfoblems 
.the opportunity cost is often assunied to equal (or at least be 
proportionate to) the market price of the resources which go into 
the program. Thfc iisSumptien rests on compethive mirket theory 
which siates;that in a pcrfectiy competitive market, each marginal 
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ictor of production will be paid an amount equal to the value of 
its Pioduct In its present and n«t be,it alternative use. Thus^n 
p^r ect competition the amount paid for P^o^^^^^^^^ 
renect the value of the opporturtities forcgon.. <Th.s gaeanstlm^ 
instructors in the painting program example m\\ be P^d 
^ount ^ual to the vdue of .heir output ,n their next best 
occupation, say, piCatiifs.) , , 

There are many prublems with the application of this theory 
Social programs do not usually operate in situations of p^r.cct 
como^titicn Most social programs involve governraems purchase 
^"SSs^ich do .^t hnve large, welWefined competmve 
nlik^ts with many alternative buyer.. For example the number 
of purchasers of eaucaiional build :ngs and P^""'"'^"^ 
limited in most geographic areas- there are ally anly a f^^w , 
school district and propHetary schools. In ^f^^ ^ 
govemmem is not a prom-ma.imizer. '(One_ of the many 
Lumptions of perfect competiiion is Pr°fit""«^"^;f " ^ '.^ 
parties.) Therefore, the government may pay economic renls to 
L owners of resources i.e., the government pays more iha^i. , 
necessary to attract those resources from altemauve uses. One 
S^ple mlghi be the hiring of unemployed workers fo 
public scrvie amploymerft. Even though the alternative t^o 
Lse workers may be very limited. ^^-^ 80-^"'"«^*,f '^Pf^f 
least the mhiimum wage and p6ssibly a considerably^h.gher one_ 
T^e social program expenditure in this case can '"^'^^^f"""^^ 
imnsier payment as tvel! as an cxpenduure to cover the 

CHHer delations from the : . feet competition model include 
mo»- y power, cxternaiuies, and nonmargmal purchase,. In 
' Ac 0-4 Ui.- sellers of resources may possess moropoly power 
.limits .aem to obtain economic rents from thc,s« o^ra,«ng 
,'pf,srams. An cxamplt might be the .mposmon ot 
.-.guiaiiom requiring the staff of . program to be res.den s oi h. 

in which L program Is operated. This may «vcrely th 
aumber of eiigibR and allow them 'o gam >a!..nes ^bovc 

their opportunity cosis. ^ 
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„ In the case of aternalitiw, im wmM pmd to the rcsourco wiU 
not renect the true opportunity cout for socicly because of 
unmeasurea or unattributable costs. An example of such an 
externality is the siiuaiion where participants in a social program 
forego employment in order to attend, i hcrc may persons w1,q 
would oth'erwiM be unemployed or employed at lower paying 
positions -that a. able id fill the jobs left vacam by the 
participants. Society in this case loses the ouiput of the program 
participants bu : gains the output of the secondary groups- OveraU 
there may be no net loss to society.' Alteriuuivdy,- if the program 
involves work cjcpericnce or on-the-job training, a jprogram 
participant may be displacing somcon.: else who would have been 
doing this work.— if this -nd disnlBccd person becorr.cs 
unemployed (or causes someone else to become unemployed), tne 
output of the p.ugram participant should not be included m its 
entirety as a negative cost in calculating ihc costs of the program 
to society.^ 

Finally there are cases where the purcli ses made for social 
programs' arc o large that they affeci the i ice of the resources 
being purch. ,d. In ihcse .lonmarginal cases the problem js 
..vaether the opportunity cost of the resources is equal to their old 
price, the new price, or a price between the two. 

Even With these problems, however, the oppdrtunity cos;^ are 
assumed to be reHected by the costs of the resoiirccs im olved. This 
occurs because no alternative method of valumB for-,ione 
opportunities offers a better solutiu. . U is nec«isary to , be 
cognizant of ihe passible shortcomnig-, of jhis appfoa. n, however, 
and to make adjustments when appr->5iriate. {The a djusiment is 
■ often referred to as "shadow.pru:ing"-inakinE an ?sf!ma!e nt 

inmfer r,aymw. .vcn (Umigh pMwmn^^ p.>? ^'^^ '"^^^ fw.^ 

gover.me«i: *h.n the W«nda,y >nd.v,Uua! ■ nd empioyn,em, the gpvct„mcn!^^«..«c 

gavtmn5*?^^l IO?^KS Oil the pafU^ipiinlS. - 
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what the i .c would h:wc been had a competitive market existed 
for the sal -f the resouKc.) Also, luckily; the problerm are not 
severe wit! . mplovment-related social prograns' as in oihn a.. 
Most resources lue baugiu m rdaiively open rnarkcf. ' 

LisUng of Com of Employment-Related Programs 

Obviously, It IS important lu utsin.guk*i. .uu.;.^ b.i-n u;;; 
being forcKonc. Wc may huvc ihc .Uudtion where the party w!m 
doing the gmm up i» not the .same the one who is re^vivuig the 
r,.ncnts. One 'of the nuv.t common mistakes oi program 
evaluation k to Ignore the cu.i, incuu b; paruw oilu^r than the 
govcrnmcnu 

Society, individual prog; am parucspauis, cnipluyui. -i^-J 
gc. .nrtient ma. . . be required u) give up resources tor 

programs. In .a.nc ca.c.. e.p..aituu. at resource. wi!l nu.n 
foregone uprxjriuniiies iar more than one group, I-or e^nmplt 
'n^ iesof imgram adnumHUaiors will b. cna. lor society a. we i 
lor thc^vcrnmcnt. lUac wili al.o be ,.ncc:, roreg^M^ wHhm 
be costs for some group, but whu:. mW be gams .or • hu 
.;.ups. For example, govci.uicnt allowance payments u 

■.articip^ftits or reimbur^cmenis paid lu cnipiuyers wih be tor 

'. ' . u ,,.,11 ... ,,..,!k. r,.duie ihc cdsl; oi the 

p....cip;uUH andJhc firm, tnvo! rhercforc, . ag.ni 
pfcscni Hcparaic li^^*^ H>r each pa- 

A. CosTH roE Si n rv 

. soc,eu-s costs for o^-ratuig f-vgraius j;;^;;:;;^ 

used np by itic prugra.n ihi. .-i ^-.avc beer: .-.r-.^ed to uSi.c. 
.,ru. us;- had the pr.,.. r,n, .cOMr. .Transfer, u. 
.oud. .uid sc;m.c.. Hi.-h as clKnu-o- m v.,/ p;iv::h;iu.s 

;-':;;;:;;:;r;-;:;"::::^^:^::::: 
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wilhin ihc soctciy ar^ mn couiu^d -^in. ihcy do not liUcr lite 
. nunc of real resources %md, only ih. : mm who use tha?i ) 
Among ihe rc^ouicc vosis f sikiuI pfog ^ Otc ioUowinj^ 

} I hc Tim^SiK^iU by AH Pc^ uuici InuilvcJ lu Uic Frogwini 

Spc^nU auu Hhould be Uikirn to mdudit ihc iuui^ i)l ihe 

UuU ihos^r v-'-o arc not cn^'iiscJ hiU iitr- m the paigran; 

a, PrajCil StatL The v ^ .houtd tnUiKlc ftir 

vaiiiif <M 'he lifiic Ihcy sfH^nd oi^ ...uv^r^ ^kjl • ihc 
iXm$\\ a ihi; project pror>o^>al; ^Cvrui^u^: ^ , ;esunK. ^^^nl 

any HU|>iTOfUve ^cjvicc:^ LCifMH^.^-CM ^^^^^ rnugra^u 

uansporcatiu..,,, iM- , iiinniK .Hiowiu,.:: payment^,, job 

ihu fc^s.MU f^scrung and adi. ";H:uranv^ MsK^ 

iUNohcti ill each v^f the pr^r^jauv 

t^enons at tht Kf -r^^jnui ^^nJ .V /.r^i^a.. Inclvu-Ni 
woiiUi be :tU pcrnv- who ar^ Hi any niui!v4:ii\^n the 
progranL Vhi^ wouU! u^ch^h: consiiltaius u> Icnal 
r'H^k:!^. fichi uperv- pcfHOfn r<:s^K?nsi,blc for 
projeci approsaS .uui rcvirw. jnd stansncinns ilivolv^*d 
I if) reviewing project rc|K)rfv \M^^n, UtKir Hatant^s ?iho I 
be aliueaicd tn pvo]->fnan ui ihe nn^c '.yKtM on ';jr 
pi vu:uinj = 

saJ.rie^ nhmild be a;, -:^^cd hi pt>-nK-mi io ihc emhc 
Hpc^. . ^tdur^i U:rHi^ progfanL ^ncUAiing ulgetar> 
tovicw, utica! acc^nHUnnv puhcy planninK. projcxi 

urn! Oic uainiiu' sinlf anU ^ochni-.i! a-asianis^ 



C hr; ^(iu!d 'include u^r n^^w'-^^i ■vhui* ^^h/t ■• 

4nJ huitdtn«s, including y>^ ^mv" u\v 

nifUKei ^' :nti^^ value -Mi mxidum ^ . im-^vmhS '-u 



^ ,, . , , . ..... 

I lull's' .U.i: ^' • ^~-^i^n^">n ^ -.1^:4 

piUi^.^nant^ m^x% ^^^^^^ i^^s. tn;M ''^^^y ^^^^-^^-j H^fM: -.^^i^^s^vu 
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Thus, to measure many of the costs requirp kaowledge of what 
would have happened to the program participants had they not 
participated, Theiest way to get this knosvledgc is to use a control 
group which is randomly seltcled from persons willing and able to 
enttr the program. Onlx this group will give an internally valid 
cstutiate. Thus if the costs are to be as accurately estimated as ^ 
prograjn outcomes, the same type of control group must be used 
to mfasure both. If projects are selected for evaluation when 
funded, this will permit the same contiol groups to be used to 
measure the costs and impacts of a program. 

The Use oj Coraroi Groups. Conuol gruupd should bi^ UbCd io 
proyide information for three types of cost estimates. The first is 
the loss incurred while the participants are in the program. While 
participating in the program, individuals usually are not engaged 
in what they normally would othenvise be doinv: Therefore, 
participation in, the program may lead to losses ^ after tax 
earnings, unempioymcni compcnsatiun, or welfare payments by 
the individuals; potential produciion by society; and taxes by^ 
government. The experience , of the control groups durings^the 
course, however, should not he affected by the program. (T^is 
assumes that the program is not large enough to affect the entiW 
labor market.) Therefore, the difference between the contror^ 
group's and participants* after tax earnings^ unemployment 
compensation, welfare payments, production, and taxes will show 
the losses actually incurred because ^of participation in the 
ptogram. ^ 

The seeond use of control groups is to determine how much of ■ 
the governmental services received by the participants would not 
have been received if there had been no program. Earlier we 
discussed Employment Service job referral services which are 
normally used by many of the persons who enter employmeni» 
related programs* Similarly, ^when svelfare recipieats enter such 
programs the counseling they receive in the program may merely 
replace counseling they would have received from a case worker. 
Therefore/ it is important that information be collected .on the 
amount and nature of all governmental services received by both ■ 
the participants and the control group. If this is known, the latter 
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CM be subtracted from the former to find che actual incremeni in 
serviws which r^ult from A program/ Thin\^ only the cost of this 
inerem§nt iri ^rvices should be compared wi^ the benefits ^|^h 
were caieuiated as^t&e differanccs between th^ two groups*- 

FinaUyi the control poup cm be used to meaVure the increment 
m program-related expenditures by the participants* Some 
propwit r^uJre that the participants incur expenses for travel, 
instructional nwteriaJSj adforms* Uving expenses and meals away 
f^om home, ete. Some df these expenditures represent added costs 
g(Sr proyam participation; others may not, however. If an 
indi\aauaJ Nvould have been taking the bus to work instead of 
taldng it to a trmning center were he not in a program, there may 
be no additional cos^of traniportatlon resulting frorh the 
progrMi* To arrive at this conclusion it is n^essary to know the 
expenditure associated with the program by the partiiflpants and 
the expenditures on these items by the control group. 

If it is not possible to have a cbntrol' group, the same 
information should be gathered for a comparison group.^AU of 
the same internal validity problems %vould occur in the 
measurement of costs as were discussedAvith reference to benefits 
in Chapter 3* Therefore, once again we strongly recommend that 
control groups be used. 

Measuring Costs for Oth^r Family Mmilwrs. In addition to the 
control group, measurements should also be made for other 
persons who might be affected by tfie program. One group.which 
is very V likely to be af^ted is the participant's family. For 
instance, if participation lowers the earnings of the participant, 
the slack may be taken up by another family member who accepts 
a temporary job that he would not normally have taken* Such 
changes could be discovered by compJj'ing the work experiences 
during the program not only of the participants and members of 
the control groups, but also of their respective families. * 

5. Tfef Implidt assumption in the use of a control group io rotoure the Increment Ln 
i^nic^ is that tht lev^I of service providfd by ths progrmni bdng evalusted d<m not alter 
the amounl of lervkf provided by other programs. In leTmi of the e=.%aniplesi the funding of 
addW^al Vo^tiensl Rehsbilhatipn programs should not re^u^ the aitiouni of the 
Emptoynieni Scr.ice or public assistance case worker aid provided to the control group. 
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Accarately Mcasuriag t^e CosJs 

^ Government accounting systems are deEigned on an appropria. 
tions basis and not on the basis of incremental costs. Tpc^ 
government cost accdtinting says that diything paid out m the 
name of a program io a cost of that program. As a result, mftny 
items are improperly charged, to a program while other costs are 
ignored. Some speciric examples of these problems follow. 
' Proper Program ^wignmenf. The costs of functions performed 
for a program sometimes are mi: charged to that program. We 
have already discussed the example where one operator of a 
program may devote staff time to job development, while another 
may use fimployment Service (ES) staff to carry out the same 
function. Onder present accounting procedure, these costs are all 
assigned to the program in the former case and all ass»gn«l to he 
ES in' the latter. Yet the same services are being peHormed for the 
program participants in the two areas. Other pxamples of ^osts 
Lfare often ii^ropirly assigned are remedial educatian. which, 
is sometimes provided by the local school systems; the use of 
scboe! bulldmgs and equipmem for traimng programs; the time 
spent by various public officials preparing project proposals; and 
the value of the services of persons on loan from business to the 
governmem or of volunteers.^ Therefore, in these and many other 
lituktions one must go beyond the costs directly assign^ to a 
program in ordef to include the costs of all of tne additional 
■ sendees provided because individuals participate in a program. 
' Similarly, if eXp«iditiii:es are assigned to one Pr^^am but m 
fact are made in part for other programs, only a Porti°n °f f «^ 
costs should be included, for instance, equipment purchased for 
erne program subsequently may be used for others (such a. 
Vetoes bought for a training class which are -bsequen^^ ' 
in other classM or for vocational education purposes) or persons 
who are hired for one project may also be used on other projects. 

voluntMfS may ^ ui<ni leisure ime ^^'™« / „ ^^i^Mh as the income 
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Inclusion of All Costs. All of the costs/of a program should be 
measured. Yet, there ftrc several items frequently overlooked. The 
first, the value of public facilities provided free of charge was 
discussed above. WhUe the use of t)iese facilities does not 
repreicnt an outlay of funds, there are still coSts involved. Most 
equipment vvdll wear oufwith use, and its use by social programs 
win accelerate the need for replacement or repair. Therefore, the . 
depreciation of this equipment should be calculated and included 
in the costs of the programs. This mil be particularly important 
for tfie often quite expensive instructional equipment used in 
vocational trailing courses. 

Even for buildings that will suffer little from the ' additional 
wear, there are still costs associated with their use by social 
programs. Other activities may be displaced. For iustance, if a 
vocational high school is used for CETA trairiing, vocational 
education classes may have to be held elsewhere or may not be 
held at all. Both of these alternatives will involve costs. Similarly, 
even when an abandoned army base is used as a Joh Corps camp, 
there are costs if the property could have been sold or leased" to 
private industry. Thus, to measure the true cost of these facilities, 
they should be valued at their market rerttal value. 

The administrative cost of social programs is another category' 
typically underreported, There is a large administrative overhead 
•connected with each program. Many of these items are inclMded in 
the outline above. The time that civil servants at the national and 
regional levels spent on planning the original program, budget and 
proposal approval, project review and monitoring, program- 
connected evaluation and research, fiscal appropriations and 
accounting, and on all other administiative duties involved m 
program operation at these levels, should be considered. IdeaUy, 
the time spent by all governmenti workers from the Secretaries of 
Health. Education and 'Welfare 'a.nd Labor oft down, should be 
apportioned among programs and among particular projects, if 
possible. Greater attempts should be made to approximate the 
• services provided 'in each of these categories since they represent 
sizeable costs which presently are excluded from most ^cu- 
lations, , . 
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Calculatliig Margintil Costs 

" Another impbrtant concept for measuring both Impacts and 
costs is that they should be i^easured at the margin. Social 
programs are evaluated for the most part in order to say whether 
they should be expanded or contracted rather than whether they 
should be maintained at their present size, or eliminated. 
Therefore, what is needed are the benefits and costs associated 
with vartoui changes in their size, for instance, what wi be the 
difference in total beijetTts and' costs of adding an additional 
person to a class or an additional 10,000 participants to a 
program? Often the cost of adding extra participants vs 
' considerably below the average cost for an existing program. For 
• example, adding one person to a lecture-style class is almost 
costl-ss, in most cases, sincfi the classroom and instructor are 
already committed. The result is a declining cost curve such that 
the marginal cost Is considerably below the average cost and the 
benefit-cost ratio at the margin is much higher th^n it is on the 
average (assuming constant benefits). Yet, generally only ovei^U 
costs and benefits are considered and not those at the margin. 
Only if the ratio of average benefits and costs is proportionate tp 
benefits knd costs at the margin for all programs , will average 
\ calculations of benefits and costs be accurate guides for making 
Iprogram decisions. The problem. However, is, how to measure 
.benefits and costs at the margin. 
- One way of looking at the problem is in terms of the total 
national program. If it has had a long history and has operated at 
a number of different sizes there may be a sufficient total nu|nber 
• of points to construct a total cost curve; the first derivative of that 
cuni will yield the marginal cost of the program. One can 
simiiarly calculate marinal benefits. This procedure requires a 
number of different studies over a broad range of-program sues 
and the real world operation of social programs does not lend 
itself to this type of analysis because most of the programs have 
recently, been developed or modified. 

Another method of calculating marginal cost is based upon how 
much it costs to add additional individuals to particular projects. 



In this situation, projects are the unit of observaiion. Arraying 
projects by ste.i and cost project wijl show the marginal cost of 
adding individuals to projects. This can be done more formally by 
regrassmg project cost on project aize to cajlculata the marginal 
cost as was done by Somers and Stromsdorfer (32), This 
procedute, howeveri has a potential limitation stemming from the 
possible corralation between the size of the program and the labor 
market conditions in which it exists. Thus, a program of 5(X) in 
New York Gi^y may yield costs that are quite different from a 
program of 500 in a small town. 

An extension of the regressioji technique was used by Hardin 
and Boms (17) when they regressed total cost on the size of the 
project in terms of total enroUeeSj the nulmber of sections of the 
course, the number of weeks it ran^ and the number of hours per 
week. This analysis was designed to show not only the effects of 
increasing the number of cnrollees but also the possiblMffects of 
^ changing other components of the magnitude of the program* In 
this analysis it was found that it was somewhat cheaper to increase 
the number of enrollees per section than to lengthen the course or 
expand the number of sections, at least in the instructional cost 
area. 

Joint Costi 

Joint costs exist when the use of resources produces more than 
pne type of output. The classic example is the raising of sheep to 
produce both mutton and wool. In thib situation, once the sheep 
have been raised for wool there is no additional cost to raising 
them tor mutftori. Or alternatively, once raised for mutton therf is 
no additional cost to having them produce wool. The question is 
**When a'resource is to be used for one' of these purposes, should, 
any cost ok asFigped to the other purpose?'* For example^ 
machinery bou^ for use in a vocational education clas i often 
used in a CETA training program. S6me people have argued that 
since the primary use of, the equipment is in the vocational 
education programs and it has been purchased for that purpose^ 
there is no need to change any costs to the training program since 
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the marginal coit of that item is zero. While suchlogic is tn« for - 
the Ss wool and mutton, it is not true for the tyProf 
^^S^ussed in most social programs TOs is — «^ by 
identifyinB the opportunity costs involved. The use of the machme 
i^S^ e^tion c4ses does ^ prevent it ^m havmg m 
mjoorturity cost in alternative uses at other times. Presumably, if 
S^Sv^s not being operatedduring t^^^^^^ 
it would be avaUab^to someone else who might rent it Su^^, 
oubUc buUdingS used for social programs have separate . 
S^^tuS^^ breach period that they are used so their rem^ 
value Se uses needrto be calculated. There appear to be ve^ 
f^ca^ere social programs actuaUy do Use «»ting ~ 
"hich do not have some alternative use. and hence a cost which is 
•SShable from the social program use. Social programs 
are, fortunately, not likp sheep. 
Examining the Costs of the First Program Partlclpanti 

The cost of a particular program.will va^y. depending on wh«n 
in^ cS^ oAhe program's development the measurement ib 
r^h most Linesses, the costs of social programswm 

dSe^^tir scale and Hfe increase A program will hav^- 
Sh^l^SiS)^ per participant when it is being d^eloped and. 
SumLT Qf pmi4^^^^^ is small. Program staff will be engaged 
t^Sn^^onneU establishing bo^keeping^st^. ^ 
proposals, experimenting pth program '^eas,^ m d ^ 
o ganizing funcUDUS. As the program b^^^n^^^'^fV^^ ^ 
arS. expari«ice is gained, less time wiUbe spent^ rl^S 
ii'dmore t'tiii CEii ^ devoted to providing ssmce., '-"""^ 

w ,wr'u"'jam. and reducing unit costs. While the 
^L'^^-^ program size,'^ it is almost cer^n 
SS:^ program wU! change. TK«efore. the ^ P^ - 
which the.MSV. M . iT Ograin are measured will be crucial, 
. particulaTly at the oeginmng cf a program. 

— > ■ i «nt leVel^iU also <,ha: u tht progrBm dovdops. ThU could 
such services m trMng imprDves, 
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To resolve the problem for a new program, we suggest that the 
beneflts of the first program participants should be related to the 
costs far later pRrticipants in the program.' While this would cause 
the costs assigned to the first group of program participants to be 
below those actually incurred, it would give a better indication of 
the long run costs of the program after it has become more 
established. Moreover, it would not lead to any delay in evaluation 
since, as . has been mentioned, the success of the participants 
shoTild not be measured until one year after they leave the 
program,' 

8. Colt data sUU ihould bo gathered at the itart of the program to make comparisons 
with later eost flgures, to find the extent of decline In eosts with increajei In the size of the 
program, and for administr ative purposes. 
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EiirctiM ^ ^ ^ * 

The following table M the m of a vocational cdiicalion prograni run joiiiily by a school system 
and amploycrs, Undar ihii program students study half of thy day in the syhyul and then learn a trade in 
an employer's etablishrnenl. At the top of each column are the partiei who fni|ht incur the Place a 
plus ( +) in the box if positive edits are incurred, a minus (-) if negative m are incurred (i.e., if the 
parly receives resources instead of loiing them), and a zero (0) if there are no costs to the party. " 
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f. Vatuo of iho siydonte' outptii on iho ;Db 

; II) from ihci govomnient 
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the Podunk V^tional Relmbiliwion offiw has f 
prabltei* CUTOts hav» flot b«fl kmplm appoiniraems for w«kly 
OTuawUng i««QM er have bwn ahowinf up laie. TWs has me^t 
tfcat tte staff they arenot being used while they 

pe qiwrtoadtd tt^^^ 

Dr, ludy, the office director* caUs a staff ranting to discuss the 
proWem. She rtplatos thai having counse4ors idle is both 
inefflefent and cosUy. She pfoposes that cUeots be scheduled for 
ip^lfliaieiits only at SsOO A.M. and 1:03 P.M. When they come 
in they wffl each leceive a number and wait until ibelr num^r is 
ThcsystemwUl be Cii-Sl coroe-first served so that those who 
do not want to walt/WiU corte In promptly when f he office opens at 
8:00 A M, or reopfens aficr lunch at 1:00 P.M. . 

Ivfs. Mickey, one of the Munselors, argues that such a system is 
not a«lrable b^ause it is unfair to the cUenw and the office 

*e«pllon afw vvaibti crowded and noisy, fnslead, she wys the 
ofrice should copy the airlines and *'overbook/* Counsdors can 
hcwwia^ see two telients an hour. To reduce unused tittie for "no 
fihows.'Three clibis should bfeiclwiuled for each hour's time. If 
aUjAf#^ clients appear* requiring W minutes Instead of 60 
mInuieSj ihosft scheduled for later appointments wiU have to wait. 
m. Judy asked ^hat would happen to clients siiU writing at 5:00 
P.M. when the ofncc ^osd: Would they be rcscheduted for 
another day? Ms. Mickey rhotight ihcy should be seen thai day 

. with the counselor recwing pveniin« paj . Tiui-wAi vetoed by Dr^ 
Judy because it would break the office budget. Mr, Fish, 
reprwehUhg the clients, argumi thai the whole discus&ion was 
making a mountain out of a molehill. He noted , the buUding, 
iighti, htfflt, and siaff would all be there and paid for whether 

. fherc were t«i or one hundred clients served on a particular day. 
Therefoje, it was no wore costly to keep ihc present sysiem than 
cither of the proposed solutions. 

\Wwt wc the 4o«s of each of the tlirce aiternailves? Bsplain 
which one you wNid choose and why you would sc|«t it. 



Chapters 

eOMBiNiNG THE MEASURES 
PROGRAMfMf^ACT 
ANDCb^T 



The daw g^thrn^ iqt ^vtimtiom of somal programs shoj^d 
provide ttia mfottnatlon to wswer ^ur lyi^s of qufetloni. 
1) Shouldm paMeiJ^pro^wn^^i^ of several 

alteri^tivetpro^aBas should bt j^Kpand^ or contracted? 3) in 
what irays «n chaog^ In the com^nents of a particular program 
!kd to toproved effiftienoy? 4) What programs Jb^t Serve ^ 
iwfimilar groups of indM^ in the 

previoiis chapter^ wiU pro the answers to these quwtions. 
Befdre suefc answers can be obtain^, howevef^ Several techniaal 
d«iSioiis must be made. These include: how to handle potential 
* impaets which occur after mwsureme how to treat impacts and 
costs which Mcur oyer time, and how^ to handle external eff^^ts 
.Md serandaiy impf rts^ 

Tneattog Pqture Imjpscts^ 

SomBi programs amy affect the participants for the remainder 
of their lives. Tlertforei to accurately assess aiirof the impa^ of 
th^ programs it would be desirable to wait UntU aU f wons in the 
pro^M teva di^*^ Obviously sjich a suggestion is ludicrous. The 

I . W« o^iiOTiWie 00 qucstlo w mMtiaaS^ im^s d w we have defWld 4iU ef teili 
perttivt Of a^liivf , wMdi omit afia «he pfo^am m impself \hm^ which 
di^iflf iNprofiW ^ him of liie imparl ifte? iht prapara 5h«ul4 tavolvc pM&st 

fej^nt** , ; ■■ ' .:■ , ,/ : ■ ' ^ ■ ■ . 

liSto^ U^ift may aUo be Iniffgpiefafior^l el feci m tr^m losgtr p^&d of 
^wii^ way N i^^si. 
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and ter ^hom. Th^«^n^,^°^^ 

^"^^^tX^^ occur. after these measure- 
require «tmiaun| wnar^^^^^^ require two sets of proje ctions-, 
menis wemade. Such wurnaiion r^uu ^ow long will the 

how vim prGgram iinpacts change ove^^ 

impactslaaS ■ .,\.^' 

!.# /^«*»pnWf• rimr It is £ion«ivaDW that 

remain coMumt, or declint as t mc P««'- P'^ ,5 .ijced on - 

an entirely new f a^gner jo _ . . example herr 

of exrasive prosthelic trmmcni phys^l h^^^^ 
,rairang^V/i.h.h=«aimng an4.h=P= os h^^^^^^^^ , 

Build seniority. , • . . , 

. those paid 16 a machine operator.' . , 

in ■■ 
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. Finally, there may be a situation where the impacts of the 
pre^m decUne as dme passes. This may be due either to, 
bbsolKcence of the sMU and training provided or to the fact that 
other avenues of advaiianaent would have opened to the program 
participants had the proram not ejdstrf. Examples of the former 
case are a number of employment-related programs whjch have 
provide traimnf for jobs with spKific employers or training m 
the use of paMcular typ« of eqiupment. The person who has b«n 
trained for the nwds of a spMiflc employer may find when he 
leaves that employer his specific skiUs are of no use to other firms 
Wring in the job mwket. Similarly, the skiUs of a persori trained 
' on a ijarticular piece of machinery may become^t least par|iaUy 
obsolete if that piece of machinery or the producUon process • 



An example of "a situation where program participants may be - 
^ven' ajump ahad of indisaduals who do not participate (but 
these other indlviduali may subsequently find other means for 
catching up) would include the individual who receives vocational 
training in high school. He may have a temporary advantage until 
" others sflbsequenfly receive such training in a postsecondan? 
school, in the armed forces' or in on-the-job training. Thus. ,his 
iniUal advantage may be eliminated with the passage of time and 
tbe use by othew^f alternative methods for advancement^ 

Given these three possible- divergent streams of unpacts over 
time, which -one is appropriate? The most, logicml method for 
'making projections would be to ^e them on the experience of 
pMticipane in other programs. If the gains from a sumlar 
program have increased at an annual rate of Spf -ent, then this 
figure could be appUed. Unfortunately, there is - Jfable basis 
for comparison. Most social programs and almost evaluations 
of them are less than fifteen years old. Longitudind data are not 
yet availBble on tlwm. Further, the few longer tem^ studies which 
have be«i conducted with comparison groups are contradictory'. 
Hu et.al (19) and Somers and McKechnie (31) found dechmng 
eaminp unpacts for vocational education after six years and for 
instituUonal skills training after Tive years; Ashenfeltef (2) found 
decU^ng earnings gains for males but reralLvely constant gains for 
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females jov?r a five-year pariod following MDTA institutional 
trainingj Borus (5) found incrMsing gains for five years following . 
institutional traimngi and lorus.and Prescdtt(7) found increasing 
earnings benefits for .men completing instkutional training but 
declimslg galna^ for the dropouts. 

^ Tim pumtion qf Impgcts, If it -is foun^d that^thg gains of 
propam p^dpatlon diGline\steadUy over Hme, may 
eventu^y^each Eero/ Alternatiwelyp if tiiay remain constant or 
incrwie during the period of observationi they mfty continue until 
the. individud retires or dies. The magnitude of the projeoted 
impacts and the resulting pro-am evaluation will be greatly 
aff^tid by which one of these scenarios is selected and by how far 
into th& future thf projections we made^ Different studies have 
chosen one or the other alternative, ' ^ 

A Possible Solution. The best method for estimating future 
impacts appears to us to be a sensitivity analysis which projects the 
impa^s for several periods with the impacts ihereasirig, remmning 
constent; and d^lining,^ A matrix can be constructed which 
presents the expected impacts undef each of the alternatives (e,g*, 
Tablt 5-1). Such a matrix wiirshow the sensitivity of the impact 
estiiiates to various combinations of assumptionsp which should 
be useful when comparing different programs, 

Asjongitudinalitfldies provide more hard data on the trends ito 
impacts "over time* the matrix can be condensed. Until then^ 
considerable^ thought .sKould be given to deterniining which 
description of the particular impact studied is most appropriate. 
Filling in the matrix is only the first step in projecting 
^impacts— theoretical or empirical considerations for ^electing the 
Vhfest estimate" must follo^, • ' / 



14, All of the long^ eviduitions have found soma btnents to persist for five yean after 

the treatment . * 4 

I ■ .. 

I 5. LongiT term proj^tidns also should take mortality and ehanges in labor for^ ^ 
partieipation rates into account. L . , 
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T«bl3 6-1. Total Jmpiett Undiir Alterhath/e AMumptloni 
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Assigning Values (o Futurt Impacts and Coito 

The banefits of diffarent typeSi of social programs may be 
, raalized over different periods of time. Likawise^ various impacts 
of a prograrn occur at different points in the tifne fpUowjng 
participation, Finallyj prd^m Impacts occur in the future (after 
the program), whepas costs are incurred in the present (during the 
prpgram). The basic question to be aris veered is **Are impacts to be 
treated equally regardless o| when they oacur; if they are not, what 
is the proper method for equating future impacts with those which 
occur now or costs which have already been incurred?" 

To construct % simple examp^* take three programs that reqqire 
ail investment in 1980 of $80 million. Program A will return $80 
million in 1981 with no further impact. Program B will return $40 
million of benefits in 1981 , $40 million of benefits in 1982, and S40 
million of benefits in 1983, Thereafter, there will be no benefits;^ 
Program C will provide benefits of $20 million for each of the 
seven yeprs from 1981 through 1987; There will be no benefits' 
after 1987, Which program is preferable or should there be no 
^ogram? ' ^ . ^ ' 

Program A is unlikely to be the choice because most people are 
^not indiffarent toward options givirfg them an amount of income 
in the future equal to the amount that they invest now. This occurs 
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\t two related rgasons. The first of these is iHime preference^' 
This is the old saying, "A bird in the hand is worth two in the 
bush " Simple proof of tWs desire to have our cake now rather 
than in the future is that most of us require that we*e paid a 
certain amount in order for us to postpone consumption. In ord? r 
for the bank to induce us to make deposits, they must make 
interest payments. SlmUarly, most people borrow money and pay 
SterestLWder to buy thingi now rather thanto wait unti^they 
have the neeessary cash for the purchase.; Indwiduals will pay. 
12 percent interest charges in order to have a new car now rather 
than saving the money and buying it two years fr^m now. The . 
individual's rate of time preftrpnce, which is subjective, ran be 
estimated by his willingness to loan or to borpw. He will , loan 
money as long as the return is greater than his tiipe preference rate 
and WiU borro^v mon^ as long as the interest rate he has to pay is 
less than bis time preference rate.* \^ 

mother way of looking at the problem of| yahiing toe 
benefits in terms of present expenditures is in terms of the 
opportunity cost of alternative projects. If a I?erson can earn 
interest of 6 percent by putting his money into a riskless savings 
account, he would be foolish to invest in a project which paid hun 
less than 6 percent. Similar logic applies to government projects. 
The government should not undertake any project which-pays Iws 
than the alternative investments that if might undertaHe or the 
remrn the resources would earn in the private sector.' The m ot^ 
private sector rates of return is important because taxauon of the 
■^Ivate sector is the source of the resources whiph will be used to 
^undertake the project. , ^ 

The two concepts are interrelated. In a perfectly competitive 
capital market, as borrowers whose time preferenM is high borrow 

"Tit must be rMOgiiized that there are factors other than time preference-such u 
pre^t^^S 2vlS to a ralny da,-whi«H will i„n„e„« wiUlniyW to morrow .„d loan 

• " " 'SS^l^Ids only if the return, from the project Jar. full, «.d ^ur«ely 
, the s~t project m certain "^^J^'J^^t 

Bttaina^e in the prlyate market. 
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^ moret the iriteic&t rata will rise'^and their time praference will fall. 
Similarly, the interest rates lenders can charge will faU and their 
tinie preference will rise as they choose the best of alternative 
projects. Eventu^ys an equilibrium point will be reached where 
the borrowiflg rate, tiie loan rate, the rate of time prefere^nce, and 
the opportunity cost of dternatlve projapts will be equal. This will 
be '*the^* discount rate— the rate by which futufe returns would 
have to be^^reduced to find their present value. The formula for 
determining the present value of impacts is: 

Bi Bi Bn^ " Bt 

Where Bi* Bu Bn are the banellts (net of losses) in years 1, 2rand 
n, respectivelyt r is the rate of discount, and t is the year. In terms 
of our example, if the discount rate is 10 percent, th^ present value 
of the benefits of the three programs are $72.^ million for 
Program A, $99 J million for Program B, and $97,4 nwllion for 
Program C 

Unfortunately p in the real world ^tha capital market k not, 
perfect. There are several major factors which prevent fche'^ 
establishment of a single discount rate. One is the t« rate ot\ 
private projects. Since the corporate profit tax rate approachiss f 0 
percent, a project needs to have a rate of return ^. imM twim m 
high as the individuars rate of time preference in ordKi for the 
project to be worthwhile for him to tna^riikc% AMth@r 
consideration would be that projects are not equally risky. An 
individual, in making his c^culation of the rate of return, must 
include some premium for the risk involved in th^ project. 
Therefore, the rate of return may have to exceed the rate of time 
preference before it wfll be worthwhile for th^ individual' to 
undertake a project. Furthermoret the rate of risk will vary amdng ^ 
projects so that there will be a whole variety of interest "rates 
charged to arrive at an equally riskless rate of return. Other 
disrupting factors which lead id a diversion of interest rates from 
the single discpunt rate are various market imperfections, such as 
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lack of communication, imperfect knowledge, monopoliei, and- 
outright discrimlnation. Finally, there is a problem of externalities 
in that the rate of return in the private market may not take into 
account all of the costs or the benefits which are derived from the 
project. This is i problem of proper allocation of costs and returns 
to a project. For instar.ce, the owners of a polluting smokestack 
may not be charged* for tte pollution. The pollution exists, 
however, and it lowers the rate of return on the project for society 
although it does not lower the private rate of return. In 
conclusion, there' is not a singly discount rate which can 
automatically be applied to all social projects. , - 
. WhiJh rate, then, should be used? As stated above, the discount 
rate should be no lower than the amount people are willing to pay 
to borrow, funds to increase consumption, nor should it be less 
than the return the funds could earn in alternative investments. 
These rates differ, however, depending on who is borrowmg ahd 
who is investing. The government borrowing rate, practically risk 
free to the lender, has recently been between 4 and 9 percent. 
Business faces a somewhat higher rate-the prime rate has been 
between 6 and 12 percent in recent years. Individuals face many 
borrowing rates ranging from about 8 percent to over 25 percent, 
depending on the risk jinvolved to the lender. The alternative 
investment opportunities also vary. The basic alternative to a 
government program is a tax cut whith will allow, increases in 
private business investment. The return on private investment is in 
the range of 10 to 20 percent, part oT which is a premium for nik. 
The return to the 'individual is typically loWer (5 tp 15 percent), ■ 
again dependent on the risk involved. Since the extept of risk is not 
IcnownJn.the investment cases, the true rate of return is not known 
for kternativd projects. 

To quote Prest and Turvty (23) * 'the truth of the matter is that, 
whatever one does, one is trying to unscramble an omelette, and 
no one has yet invented a uniquely superior way of doing this. 
Therefore, the most logical way to proceed is to consider a variety 
' of possible discount rates aiid then to test how sensitive the 
' analysis is to each choice." Rates of 5 percem, 10 percent, and 15 
percent would appear to represent a reasonable set. They should 
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' covar th6 ranp of time preference rates. For most socifl program 
comparisons the relative values of the impacts of t\^o ^ogr^s 
wiU vary littie with the choice of discomit rate. In somei^liuationst 
hbwever, substMiti^y different results wiU occur.' 

' If only a single discount rate is to be used, it would appear that 
the consensus is now at a rate of 10 percent, particularly for 
discpunting society's future benefits. This rate roughly approri- 
matis the return on private investment when a ^mall allowartct is 
made for risk. However, other rates should be used as well in 
order to demonstrate the sensitivity of the .analysis to this , 
assumption. ' - ^ o 

AccountiDg for Exttrnalitfes - 

Up to this point* governmental and societal impacts and costs 
basically have been calculated by aggregating the impacts and 
" costs associated with the program participants* We must now be 
concerned with external impacts and costs—benefits and costs 
accruing to persons who are not directly participating in the social 
prograni, ... ' 

The question may be raised as to why these outside individuals 
^ should be considered. The answer is that these individuals may be 
affected in ways that completely offset any value of the program. 
For examplep it is conceivable that program participants are placed 
in jobs which would have gone to other indiitf duals had the 
program not existed. Further,^it is possible that the^ program 
participants rep^ed individuals with identical characteristics so 
that there is no net increase in employment in the .aggregate or for 
any particular group in society. Alternatively, the program 
participants may replace other individuals^ who hold poUtical 
power with the result that the program becomes politically 
infeasible because the latter refuse to allow the program to exist. 
Similar issues revolve around the question of who pays the costs of 
social programs. 

^ ^ g; In terms of our axampla, a 5 parccni discoum rate will cause Program C to be 
preferabU, whereas Program B has the highest yisld using a rate of 10 percent. If an 80 
percent rate is used, ^pgram A is to be the best of the three aUf rriatives but no program 
would be preferred to all three. * 
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There are two typei of external benefits and costs. The first are 
real extemaUties. They involve the creation of additional real 
productioii (real goods and services which would not have existed 
before), or they use addiUonal real resources (and therefcrt they 
reduce the amounts of goods and services which can be produced). 
The rwl external effects are particularly important when one ii 
discussing the impacts and costs of social programs from soaety i 
point of view. 

There are several simple examples of real external effects of 
employment-relay social programs. First, there are the effects on 
other members of the participant's famUy. For mstance, an 
individual may be provided with services which increase his 
earnings. This in turn raises the famUy's income. With the increase 
in family income, the spouse and/or some other, family member 
may reduce the number of hours they work. Whereas the family 
may be happier with this aUocation of labor market effort, the 
increment in production is only the change in the, output of the 
total family unit, and this is less than the increase in the 
production by the program participant. The loss in production 
which results from the reduced work effort of other famUy 
members must be subtracted from the increase for the participant. 

Another example^of effects which are external to the individual 
participant but are internai to his family would be the mnuence of 
the program on the investments in education 'and human capital 
• made by other family members. For instance, the higher earmngs 
which may result from a social program may be spent on t^e 
program participant's children's education so that their productiv- 
ityds subsequently iricreased. There would be imergenerational 
effects of the programs which increase society's output of goods 
and services, although over a relatively long period of time m this 
case^ 

A third example of real externalities occurs once the program 
participants are employed. If they interact favorably with other 
workers in their place of employment, the program participants 
may either transfer skills to the other workers and/or form a work, 
roup which operates very well together. The result may be that 
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the other workers receive higher earnings due to increased 
production as a result of the existence of the employment^related 
program. 

In all three of the above cases, there are obvious effects on the 
real output of society produced^ by workers other tnan tne 
participMts in the Social program. There is another class of 
externalilies called pecunrary e:^ternalities. In these cases,ahere is a 
redistribution of income to individuals external to the program but 
there is no changejn the re^l production of these individuals. For 
example, if a large number of workers are trained for a particular 
occupation^ it is likely that the wage rate will decline (or not, 
iiicrease as rapidly) for all workers in that occupation, including r 
those who had been formerly employed there. Since there is a 
change in earnings but no change in the output of these other 
workers, this is k\ pecuniary externatity and not a real one. Society 
has not' lost any productfon from these workers and, therefore, 
societal impact of the program is unaffected.' It should be pointed 
out, however, that this redistribution of income from the 
pecuniary externality may be extremely importani to the 
individuals who have suffered from it; thus, ihtiy may aut^.npi to 
thwart the continuation of the program / ■ 

There are four other external effects which are very iniportanl 
for employment-related social programs-^tb" displaQement, 
vacuum, 'substitution and complementary effecLs. They may be 
either real or pecuniary externalities, depending ' on the 
circumstances. 

^ Disptacement ^/Ms. The displacement effect occurs when a 
program participant either causes a presently employed worker to 
lose his employment in a given firm or, what is more^Iikely, causes 

9. The governmefit impacts ubo may nui i>c wh^ingca. ...uiuy^h tc^c:.Uv. ; i!i 
decline ftpm the othgf worktrs.^hU should be offset by higher iiwes paid by other factors 
of produeiion, ^ 

10. One can argye iharW reluctance of n}any groups to accepi equal employment 
opportunity prograim b an attempi on ihpr part to avoid the negative pecuniary 
cxternalill^ ajsociaied with these actions, A prime example ii the building trade unions 
who refused to admit large numbers of jour neymen or. for many year^. to wperatc wnh 
government apprtntlceiihip programs for blacks. 
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m ^ . J ' ■ 

^me individual who would have be^ '^^^^'^t^^ 
.to get a job b«ausc It was taken by the propam. P^^'Cip^u 
rait ov^ eknple of tb,is is an employer who hir« progrwn 
^ef,^« and ke*ps them for as long as he can use th«m ai som 
KdS mte. but then iepla«s them with other workers who we 
subsidised This of course is a rare case, a .moi« uomu^^n 

is the situaUou where the progrwn 
somewhat «cater sUUs due to the propam than,do other w?Ofkers 
«^^e^vSlable in the labor force. The program participants wjd 
N tod . rather than the other workers b«auBe the arm 
^LSly wiU reap greater proHts, by hinog these mare , 
productive individuals." ^ 

Whether displaament causes real P^^^V 
depends on how weU the labor market operates, parucumrly h^w 
the labor market approximaies full employment e^uiUb- 
riSm'Ld^ the time ^riod tSatJs involved in movement Jr^m one 
. S^^im to the n^t. If the assumptions are made thai there ,s a 
SSl^loyment equilibrium at the start .he P W ^ 
woikers ate being used to their fullesl capaoties) ^^^^"f^ 
fMid adiustment to a new fulUmploymcni ^uihbnum. then 
hS S^^o were displaced would be able to move ,n^ 
ofe:upaUons winch at the margin pay tl^sa^^^^ 
rate as they were formerly earning. Any deviauon m.the 'nc^J"^ 
" output for .ocieiy from the incronent ,n the earnings of the 
orS" program participants svill depend on the ^^^^^ 
aSmem to the new equilibrium. In the case where the 
ad u men is instantan.OU.. displacement does not lead to real 

externalities (assuming the adjustments are sm^ 
that the wage rales of the displaced workers w,ll be 
unaffected), • . 

wiU riteu ih« diWldty of the dtmand tu^ ^ horUonui ihcft will bt 

diipU«iiiraUiflhed«nindcur^e»»hon«nw!or»^ , 
no djiplawment. 



^ % liowever> ihcr^ is than fuJ^^nip!oynii:m so ihai ihcm aic 
iMs^ iovolved iri poviiig ftom orie^uilibfrniri lo ihe mm, then 
thtrc msy be wtema^U^ involved in displac^inent. ^For 
kmmm^ ItwQtkmw^imm mimnpldym ihty are bumped 
down from one M^patioo fjo IM nexu this ^od of 
unraiploymetii invoiv^ a cm lo b^^ou^i; um^^ 

worf^tn aia not prc^udng and sfrw^s. ivioicovarj lo Ufe 
Oftdit that pme of thf" displaced vi^orfccfi do find 
ottpIpymOTtp thfir teek of eraplojraeru must be con5idpfed as 
nt^tivi impaet of the propamrThe crucial ^tsiipn ml^jtoftuftiJDg 
dispiaieffleiit tffMti is him tpng k lake^ for ihp adjuimehi^ 
prb^s to dcwr Mdto.what fVfHMlspla^ workers gre l&ely lo 
be undDpIoyedi It wo«jId appwr that the high'tf ihn^rate of * 
^empioymem^ the tdttgcr mh he the perioti -difrins which the 
displaced 'wi^ff 5 are unempjoyed and the larger ' the pwl 
externalitii^, ^ ^ * ^ '/ ^ 

^acuud Effe^is^ Thi^^^uum effect U m some ^aspects jhc 
reverie of the displacernemeffwtp If individuals w^o pmldpate^ 
insoci4prpg^E V^uSd hnvrheld^^ ihey n^t b«n in^e ' 

proffram^ the effect-^f progranf participation is to redu^uie 
supply of labor in tho^ octfupatlons whifh^l^ left. This might 
^nereaae the wigrmte in the occupation and^aitrmci workers ffoi;^ 
other occupatidns the riwult ihat the reductlpn in emplc^ent , 
in this occupation wowW be \m i^m^it number of persom who 
Aiti it 10 enter the i^rogram.*' Again, ^leiUfr tbre are real pr ^ 
pecuhiary extVAfttfUes Ml! be a funcuon of labor ■market 
conditions. There are HKeiy io be real extewialities if there is lesi 
than full employment which, allows iome workers to take jpbs 
when they formerly woutd Have beerrunertployed or underem^ 
ployed, This meafw ihai th^^ppo^umty cost io socieiy in term^ f 
lost DfOduction wiU be l«s rtian the output not produced by ;hc_ 
pjiffjRpsrits whilft ih^^re in the nro^ramVJn cnntrasi, the 
Wacuum effect may be n^ligiblci or ^lave no impaci ui all }f there is 
wU employment and th^ number of worker s involved Is pteU. * 

lUpply curvti. Tttfft will be « pants) effe^ri if t>nih ojfvei %\t\p^ jrnhi! m-jf Wfl! dir^i^n*, no 
If ihed^njwa cum U lailcaJ or Ihe supply c*Jfvt hmUmiA\, andit lotil effcM H 



in mikmv >Uu:t Chaa UK-ic wcHcn tin' pr0^.:.^r:. 
.,.u ,,vuirtuf u ft^MMt 4r.::,r Mu.n Of ^Mi.ca k-vd wm^im^ 

y-'- ............ tcVi,:! 4rmii:^^^' 

- ■ -^-^ ■ - ■• . 



ihey can find Liter coraparablti work. casUy^ the iosscs will be 
minima! to ijrms of ouipul foregone, However, shere wi!! be 
swbstafltial /cJil external ef£c«s if thcy remain unerBployed for 
iong iwriods. 

Complemmfao' Effim. Finally, sheru may real vomwa 
mmwMm aiid sodal gains tmn if thcr^ is no uncmpioynwn?, 
wh«ii trsc pia«i?icw m uieprograni partici^niui -il-iu Um^ iv> oUj.;:., 
beimg biKd. Technology. Is; such tlmi jobs arc complcnwiUttry in 
sortttt industries. If one job is vaCMis, olhct |obs v4uc!i are 
dcpsndeni upoti U ca.r5noi be micd. For exarapii;, a shortage of 
computer programmsra will Ifmit the use of wmputcfii and the 
«eed for keypunch opcratora. In ihc« cases, tht improvcrntivii in 
the ea?ninp-and crnplpymcnt of tn- i?rpsi.n3i j),tnicip;ua (whc 
be wines a computer op-eraior , in ihc csa,mp3c) undersiates the 
hemfm which socieiy receives. The complementary- workers 
topunch operators) have liighcr uarniiigs or more employment 
than !hcy would hfivct these, too. are benefits resulting from slie 
orogram. The eMtem of such cofnplcmcntary effects will be base*! 
on th« lafror market situaiion apd lechnology in the Iridustrics in 
which program pnnldpants arc placed. For the complcnienjary 
effects to occur there nwd to tw bottlenecks, which arc most Ukely 
to oc«ur only m flic economy npproftche;i full employment^^Ji 
would appear that if the part tcipanis ill i m ops;«jog.'i 
dlsp5aciiig other worker;; thcrfi are Iskdy to be no complementary 
effccfs, . ■ 

dSfficuit to 'measure, Tlicy.rcqtnre identincitUDn of the pm^ 
aiitside the progtam ajiti the comparison of their situations in the 

pr«enc« of ihe proBiain with what svould have h-appened in . its 
absence. For mernbefs of the pariicir-nts*, families, such 
isrowdures are relatively tasy.™.thtf fiinHisc:; of controi or 
mmpmhmi mmp mm^m wUl repr««»! ihc suuation wiihoui 
, the progranK ■ 

The other csternaWie; do , noi. kna tnemsftves to tttsc' 
mamifdmumi. Theri; ss i c way !o idenyfy . posiisveiy ih« 
individuals who woukl have been hirtrd wtrv the prograin 



pa^u4pants not hired first, nor is there a svay to idemify the 
individuals who arc hired when program participanis do not apply 
for a job that they would have rilled were ihey not in, a program. 
The best thai appeajs possible is to realize that the fxternaJities arc 
dependent on the level of the cconqmy and the si/.e of the 
program, to examine them, carefully, and to present some 
alternative assurnptions about the displacement, vacuum, substi- 
tution, ajid coinpiemcmiitry efiKci!,. . 

Secortdary ImpiHM. In the preceding section the discussion 
concerned direci 'effects of social programs on persons eKtcrna! to 
the programs, there may also be indirect effects of the spending 
for the programs-~thc secondary or multiplier effects of the 
nmsTam.i. In its simplcsi form, the logic runs as follows. Social 
prograins represent an increment in cxpendiiures. As I'm monies 
"for the program arc speni by the program participants and tlic " 
program' staff, the result will be greater income for other 
individuals who receive the funds. These individuals will, in turn, 
spend more. To the extent that, resources ar# unemproyed. the 
expMdiiures will cause increases in prnducfion through the 
muldpUer effect. These secondary effects could also be included 
among the impacts of the progfani. 

Most evaluaiions, howevcf, ultimately concerir'Th«, choice ot 
alternative program cspenditiircs or a reduction in ta.-^s." These 
alternatives will, also have secondary effects, and it beeomes 
merely a question of how the multiplier wiJl be called into effect 
and not whether it wiU he appHcable, Since all programs wHl h^c 
a muUipIier effect, there seems no point in conlidering si directly. 
One need not multiply the benefits by a fixed factor (such as a 
muUiplier of two or three) ^nec all programs will presumably have 
.such a niuUipiiet.''' ■/ . 

' i^'Xr^- OTi be m^Jt urn mc li.s m^mh-:t '^W ^tp^ns m ihf 

tvpj of «!-ndltuir imJ ho« H Mtif, t.K.. "rt,n.n!! r»«);n.!i« in>(im M:c 

pjftrtical and, ihuctae. iiity pwi'iifcSf . ' 
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While the usa of an overall multiplier applied across the board 
does not appear useful, the geographic: distribution of programs 
niay cause them to have diffgrf:nt impacts because the degree of 
unemploymant varies = considerably between areas. There will be 
pockets of unemployment and^depressed regions even in periods 
of relatively full employment nationally. Spending in areas where 
there is high uneihployment^^.U produce- greatef real production 
guln? that! will equivalent amoiuil^ spent in areas of full 
empiayment, where the additional spending will only lead to 
higher prices. In the case of employment-related social programs* 
the effects are probably much more localized than the eff^ts of 
alternative development projects. Most of the expenditures for the 
program would go toward the purchase of services and income 
maintenance payments rather than for capital goods. Since the 
expenditures for service and income maintenance would be spent 
in the area in which th^ program occurs, most of the program 
expenditures remain in the community, at least in the first round. 
Of coUirse, as the income is s^nt on manufactured^ goodj in the ' 
second round, it will most Bkely move outside the geographic area 
in which the program occurs. One would expect t& fmd, however, 
greater Increases in real productivity in are^s with high 
unempk '\ent, than in those where there is full employment* 

' \Vhiie it is possible to indicate the direction of secondary affects, 
as was the case for extenmlittes, it is impossible to measure these 
effects with certainty. We,, therefore, are face^ with the situation 
wh^re wc must attempt to estimate the external and secondary ^ 
effects of social programs whenever possiUe while, at the same 
time, realiiing that we are probably able to touch only the tip of 
the iceberg. It is because of our ignorance of the iubmerged part 
of the iceberg that we should be very cautious in using the results / 
of our analysis* ' 

Maklfig Progriim D^blons 

In Chapter 2 wc outlined a scries of many different type^ of . 
possible criteria for measuring program impacts. Any or all of 
these criteria may be considered to be important by a decision 
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been examined in the analysis. 

the ratio should show the present value of each, i.e.. 

n ' 
2 '(Bt/(1 + r)') 

Present value^fbenefits ^ i^L__„_— 

1 = 1., 

where B. and C, are the benefus --i "S. for ea* Vjajjr, 
terms m the ratio w'"' - J. . .u, of resources to 

SthS'IhT ;ese„. va.,e or .he econo.jc ,«ujns «c^^^^^^ 

' weighting is necessary. , \ a 

Whereas .he benem-cos. ra.,0 P™?*[^ ~ X's*U 

- doll.™ (such "''t"":;" r;zri ..r'aLi°»H>=«p"»'S"'"^^ 

sc.l.), *"<P''"l™»"'"»'-"°-'*""""'°. ■ 
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recovered, but in addition, further loisis are incurred after the 
conclusion of the program. On the other hand, if the numerator ii 
' positive and, the denominator is negative, the program not only 
generates successful outcomes on its completion but also provides 
more t^sources duriiig the program than it consumes. If both the 
numerator and denominator are negative, the nW gains are 
generated during the period of the program while net losses occur 
after the completion of the program. Such a situation is highly 
imusuaL 

A related .problem with benefit-cost ratios is that one must be 
clear as to what constitutes a benefit and what constitutes a cost. 
Earlier, we suggested that costs include those changes which occur 
during the course of the program and that impacts arc any changes 
which occur after the end of the program. However, because 
different programs occur over different periods, it becomes 
somewhat difficult to apply the dtflnitions of benefits and costs 
equally to all'prdgrams. Yet, mis can become crucial in the 
caleulations of benef it-cost ratios. For example, tome employ- 
ment-related . progrlms provide on^^he-job trmning or work 
experience where the pwticipants produce useful output while they 
are in the program. The magnitude of the benefit-cost ratios will 
diffe^j depending on wTiether this output js considered as a benefit 
and added to the numerator or a negative cost and subtracted 
from the denominator,' 

A third problem with the benefit-cost ratio is that it requires 
that A fate of discount be established to find the present value of 
.the benefits and costs. As discussed earlier, the selection of a ^ 
particular discount rate may be subject to question, particularly 
since! the choice of rate may determine the magnitude of the 
beneflt-cbst ratio. However, one does have the opportunity to 
make a sensitivity analysis to find out how the benefit-cost ratios 
for different projects vary in their ranking if different discount 
rates are used. When such differences occur, another combination 
of benefits and costs may be more appropriate. 

When the numerator or the denominator of a benefit-cost ratio 
is negative or there are iubstantlal problems in differentiating 



bineflti and costs, another combination of impact and^ cost 
measures-^the net present v^ue which gives the ,«t 
gain ftom the program-is usuaUy more appropriate. This may be 
expressed as: , 

We preient value of benefits minus the present value of costs 



or 



% (Bt/(l + r)t)= S (Ct/(l+r)t) 

It shows the present value of the additional resources which have 
been gdned after costs have been deducted. Therefore, the net 
gresent value must exceed 0 in order for the program to cov^r Us 
costs, ' . , 

^hlle the net present value combination solves the problems of 
negative wsts or benefits and the definition of bencfiti and costs, 
it. too. suffers shortcomings. These include the poWem of 
choosing a discount rate.- In addition, it should be obvious that thex 
net present value can only be calculated when both the benefit 
and costs are expressed in the same units, as is also true^for. 
benefit-cost ratios. One cannot subtract apples from oranges or 
doUars from units gained on a satisfaction scale. Also, when 
comparing altematN programs, the net present value is 
appropriate only when the expenditure, on the alternativ^ can be 
the same. Otherwise, there will be a tendency to choose the^large 
project. For example, using present v^ues for both c^ ^ 
A may have a cost of $10 miUlon and a benefit of $1W mdhon 
wlSe Project B h^ a cost and a benefit of $100 and SMW miUion. 
Sri^iveiy. The net present value of Project A will be $90 million 
whL that of Pr^ect B is $ 100 million. Project ffa^tars supenw. 
Yet, if Project A could be expanded it would be much more 

trentment of mgiUve beniflts and coiti'. m observed with the benefit-cost ritio, r«ppear. 
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A third measure, wfien feoth the numerator and denominator 
are expressed in dollars, is the internal rate of return, This is the 
annual discourit rate which will equate the total benefits and costs. 
To estimata the rate of return the following equation is solved for 

rt . ^ " ' ^ ' 



For a program to be successful^ it must have a return graatar than 
0, to be better than an alternative program, it must have a higher 
rate than does the alternative. 

The, obvious adyptage of the internal rate of return approach 
over the others isihat it does not require an explicit discount rate 
and thus avoids the problem of choosing one. Also, it makes no 
difference whather a particular changa is labeled as. a negative 
benefit or a positive cost. The internal rata of return suffers, 
however, from other shortcomings. Most important of thesr is 
that it cannot be Calculated without the use of computer facilities. 

Should a Program be Continued? iThe answer to this question 
usually dapends on what alternativa programs are available. 
General agreement should exist on tha discontinuation of certain 
types of progTMis. A program should he ended whan no 
redeaming features are found after consideration of all^ criteria of 
success=where all important dependent variables are measured 
and: 1) no benefit-cost ratio is greater than 1; and 2) no cost- 
effectiveness ratio has a positive numerator or a negative 
denominator. These criteria will be met very infrequently if only 
because it is usually impossible to quantify b1\ of the"dependent 
variables* Therefore, the program decisions must be based on a 
comparison of alternative programs, ^ 

Comparison of Alternative Programs. In vary few cases, one 
program is superior to another whan compared on all of the 
criteria that we have suggested. In these cases the course of action 
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is clear- the superior program should be expanded. (This is based 
on S'awS^ that average benefits and costs are positively 
^^^S at the margin^ More often. one finds a^ogmm 
wWch is superior in some areas but inferior m others. The ^ho^ 
^^ranfexpansion and eontraetion under tl«se 
d ^ nds upon the preferences attached to each of the ^m^ s^^^^^^ 
examole a skill training program may be more effective than a 
■eS^^ion prolam in raising the earnings and ^dumnij 
hrunLmpSent of the participants, but the remedial education 
^^^3a^ to greater personal satisfaction ij^ovemen^ 
in race relations." Assuming that only one vrof mijm b^ 
exoanded a choice must be made as to which is mo« 'mportam 
S^ed ewnSg and employment or psychic and behavioml 
mcreased earning ^ importance 

^ic^eights should be explidt so that others who have 
different values can also use the analysis. 

There are two strategies which may be followed in assigning 
rellive w^Ltrto program impacts. The first is for the decision 
n^v^^he evaluator with the weights of vanou. 
bSts befo^ the evaluation is begun. The evaluator wd then 
SSStti^measUres of success with non=«roweighua^ 

• his findings to arrive at a ^^^^ ^^^^^^^ 

nm«am effectiveness. The advantage of this approach is that i 
Sef^als deemed irrelevant (those given no we^ 
ctoes noi f therefore it may be more economical and 

:£^";of:L« is .ha^ .he weights assigned to 

thwefore, W pr««t the r^^^'f^^^^^'^'^^'^J^iatBk^m fo|m of ratios of 

. the inerenwnt in a nonecononlic beneHt _ j .aiisfagtion by 2 points on 

instance, the retnedial education program mWin««"lP*"^»' .-^^ „tio then would 
, Seof20«tacostofSS0toacWeveth^m^.n^^ 

^ 1^. T>^«f V^'^S'^mfSSS'Skcr to^Snreiativ. weishts 
S " cost-cffectiveness ratios. • 
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impacts differ among'decision makers and over time. For this 
reason the alternative approach usually is more practical. 

In the second strategy the evaluator calculates the bencfit'^cost 
or cost-effectiveness ratio for every impact which might ; be 
relevant for each program being examined. If consideration of all 
possible impacts is not possible because of cost or other 
limitations, the calculations should at least be made for all impacts 
which are thought highly relevant. The ratios for alternative 
programs can then be compared in a single table,- - This procedure 
allows each decision maker to assign the weights that he believes 
are most appropriate and to arrive at a decision of overall program 
value. If circumstances change^ the decision maker can redefine 
the weights he wishes to use and simply recalculate the relative 
performance of the programs. The weights should be determined 
independently of the results of the analysis; otherwise, there is a 
great post-analysis temptation to find the weights whigh will make 
the analytical results conform to previous prejudices* 

Improving Program Efficiency. The same procedures could be 
used to compare the benefits and costs of the components of a 
particular program la order to determine -the most efficient 
combinations of components; the multivariate ainalysis. proposed 
earlier includes the effects of the presence, the duration, and the 
quality of program components on each of the meksures of success 
and cost. 

There nAy be a component which has no benefit-cost ratio 
greater than 1 and no positive cost-effectiveness ratio for any of 
the possible criteria. Such a component prdbably should be 
dropped. In some cases, however, components have to be treated 
as complementary sets. For example, diagnostic testing by itself 



20. Such a table might take the following form: 



Benefit-Cost of Cost-EffiCiiveneM Ratio 



Criteria of success 


Program A 


PrO|ram B 


Program C 


Increased production 


3.0 


2.0 


1,0 


Reduced unimployment 


1 week^Sl.OOO 


2weeks/$l,000 


3 wcfks/Sl.OM 



Increase taxes 2,0 2^0 ' 2,0 



13. 
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wiUn^aUeno improvement in the f t^iJc " 

useful counseling may be «'^"=^^>y^f"^'*'',^;' Effectiveness. 

however, that ^ompm^mn^ J^^^ 

depending on the cnter»or^ of success Once ^ benefit-cost or 

for each component can be -"^^e mc 

cost-effcctlvenesi ratio for each of the criteria 

the choice between components. 

MatcHins CUenis an. Programs, ^^tid^i??"^ of , 
analysis and presemation may be used t° 'denu^ ^h. _^ 

various Pl<^^^^^/^^^J!^% ^?Ucular 

"'''''"nf no cnUa "Sn The multivariate analysis would show 
groups of potential cnems. ^ „,^Hnre differential success or 
whether programs or component. P^^f""^^^^^ data, 

aoss'.program„and cros,s-compon=« compur,so„>. 

Stop, for EvaluaUng the Impact ot En>ploymcn..Rc.«.cd 

Programs: A Summary 

L. -^^^n, nf ail emDloyment-relatcd social 
In our opinion, the impact of ^^^P' ^y.h ^valuations 
programs should be ^^^^f'^^^^^ o^'ational and 
■ Lginning ----^S^^'^^.S^sc^ion provides 
cominums on a periodic bas^ _ P^^_ ^^^^^ alternative 

what may appear lo b= ^ .^o^^^ of employments 

approaches to measure '^P| ^.,,,33^ and to 

related programs. Here we wish "4, ^ j^,, for measuring 

outline what we feel would be a us ful P O^^^ 

program impact. The steps include the following. 

1. Examine the components, cllems and operating conduions 

of the program. ) ' 

■ ■ „, Detern,ln« what >,s .he nature of U» P'.os;™;,^!!"*' " 
is, and with what other programs ,l is comparable. 



b) Identify who are the v rogram's clients :nu:l iunv they arc 
recruited and selcctcK^ 

c) Determine in which iuoor rnarkeis the prugi asT^ operates, 
■ who arc the prognun opcrntorn, afid how the prngrani is 

supportfJd. 

2. Determine the possible impacts of the prngram. 

a) Determine the goals of the program perceived by thf 
commiBsioners of the evahjaiion. 

b) Determine the gonis of '^'sij^nificant (Uher^^' wh»'* tuny 
make use of the cvaiuaiion— program oparnjor:;, 
poUticians who originate or review such prc^graniH, other 
evahmtors (possibly through pubhcauons), nnd proHram 
IMrticipant*;. 

c) Review evaluations of siniilnr prognun^; to dctrnninc the 

significant. 

d) Think of all other possible impacfs. 

e) Identify the impacts with the party being a^ected— 
society, government (or a particular unit of gov^r nmcnt)i 
einployeis, parlicipaniSi and participants* far^ilics. 

3. Establish one or more measures of each impact. 

a) Use eKisting scales. and ins/ri.unentH%hcn possible. 

b) Apply marginal productivity (heory lo estiinatu hociclyV^ 
gains from individuals' earnings gains 

c) Be concerned v^ith the validity of the nieasiires. 

4. Determine which impacts should be measured. 

a) FoJlowthe principle thai it is better to measure too many 
impacts than to omit ont whicii is important, noting that 
priorities for programs change with economic conditions 
and ooiiticians. 



b! l Aiiuinatf only fhosf mn.' - i ;^ 
■ n>casun^ winch uv.ua ...,tn.nm.. 

Vhi.h no Mf^nificant .Mhs t :M,f .^,m .. 
,xicrnal .valuiitv m .r-timainn: .lu ■ • 
nrograni 

Note the tl.car-:u.m..n.t .xu .... v.h-inv 

.Mf random U->S!j- lui .cui i> , 
:.ln,:rnaMv.- .s lar as na-Mn.. a. : ^^^^ ; _ 

: ; nbab!v ihcTCt.rc'aaoti -a- - ■ ^ 

y less a-u'tul lof aciiC(,di, :.i ■ 

a;rcct>nn ^md niuaa-iula . 

iixpcctcd ruires. 
. Ue,crnuneUKPOtama.ic.--..-na.ann.ana 

- . ■ ,,.„,, a, .ui.-- as -he soir 

/ .u ;\void the UM- Ml ^ ■ ' 

nit-asHrt' nt coa 

, , a n.-pprtnnnv .u.t measure 

L,n.e.ar,i.Up.odu.uvUv,U.nvu,a.nna.cM,na..a 
' of society's opporuinuv cc^^ts, 
uMuw..*;., ' - ,,,,.,,1.., nt ROVcrnineiiM, ^.m- 

, ^ . „ ,. .1,,. ,„a sncrcniem m cos\'^ 

7. Apply n dc-ai:n wnui. ^ 

s .rd -.MUr^a fUmips '»nuu!d 



ly|H;'> and uin>^^^^ ^'^''^ '^^^^^^ -hwu!;: lu: 1';"^:!^%.;^.} 

from different^^:^7fd v\^ivcv^^ m r-v.^h--, -is-.^^ 

than c^iabhHhn);- r.^-^^ d.:^ ^ ■^;ic^"uui ^v^i-ltir H m-^ 

Cust data 'Mi^ :■: ..'ulc.::cJ rtn.;^ r:-Mr:;P ^ hvccfnt 
Uiily <)|>cfable u> avon.i inajal n;w:n ^a^r.^vc dyr 

luHiue, lengib, tind c^: iU! ■ -r-u:--^ per fnnnASiof each 
'parncipanr . ■ . . . 

indiaumg chun>^c-^.n h)- ^knou l^d^c ^uio ^-kUB ana hi> 
\\\n\idc> luward^ the pfogr^un. fot ^adl rur:>on ^^1iO 

canducced dctttrniine rjjvni> (or h^lymg tnc 
proicci ana thequah^N .^nd njN:rv :u-^ 10'' k-avc - 



I) Urn mutiiwiialf .mMm m mm Jot -Uifto«w& 
Ummi iMt mnml ot mmpmimn powp mOTbtri and 

aimim tmm prognui^, tor poups of pBitidpMi^^ 

for wii=0Ul CTiidBif^n^ yiidff whkh 'the prapOT 

a)Fmj€Cl tilt toj^as for «sp*MM du^suoa 
todade any «pecied growth m dsclta^ In *hfnL ^ 

p)DiMroum lilt impacts Ma co^tiv \mi Ihdr piwisl 

secondary tmp-ito. 

la Make cakiiteilOM aJid discu;:;!; ette liUidsr^::- . ^ 

^intfrMi fSie of retymj tor Mch mtpaa under wmWrr- 

c) Pttmni mx^MM^ cd^^m^^ for fpgiisfi^mi pmm ^ 
dk^*, tor difftnaji ^m%\i^mmm^ m pr^pa^ romp#^ 
n&fi^, mntl for pfojeci^^ op^tmm^ yrtckr dlficrtig 



m 



4) Indl^ie how ikm^. mki^mmm are mbjmi to tht. 
Bmwmptiom made in ihe csUmauq^ praaess. 

k) Pt^m youf hmt ^timatc for Uie effctilvfioei^i of cha 
0 SiiM^t topr0v€m«ts ta ptopam efficiency wMicb 

Thw iferw basic UmiiaUons on jhe impact irvalmiiioit 
oi^tliods hpY^ pfesopiied which shoOld always k k^pl m muid. 
Firstt ibt aaaJpIs wUI be based ooly on sonie of the critcna of 
su€^^ and Ofiiy oil some of ths msis of &mml pfQgmm$. Our lisis 
of tapactf and c«is are adfuiuedfy Sn^omplete. Evert if mom wot 
added to each IWi, conslranis on lime, funds* and abiJky would 
lirnit the materials which could be considcrtd and Jmpoitaflt 
mmmm^t beomiit^d. Mot^ mpomm, however, sv§ consider 
oiily tbose crileria of suotsj and costs whsch can be rneasurcdi 
ifTior^ thorn which mmoi be measurfid. For these r^sons the 
judgmtnis which are inade abowi iht prograTO^ componciti^. and 
paitklpants may not always b^ -'righi.'* 

Second, wm for meaiurable criteria, cotmci evaluaiians of 
forial proR ns mnnoi always m designed. Wc haver argued iliat 
mnffom **^^*^nmCTf to mmtim^nitii «nd control croups will 
ptmil couiiderably more awurai^ and irustworihy tvBJtiaUons 
itian will oihar pnethods of mmpk selwiioii , There are uB thtmw 
to wterasJ v^idiiy involved, however, whsch may Umii ihe 
gaieialteabllUy of th^ findinp Not all variables which could 
concavably innuence the ouicomo$ md co$u of social progranis 
mn br mdud^ in any anaJpis. Thctrerofe, an ob.^rvird 
iftoiionshJp may bt* diie to some oibcr fuctor which is correlaicd 
with ihe wo vatiabte bang examined. For cxampie* h may hn 
lound lhaton4hHob ijralnjfiig has higher bCT^fiMOAi faiios than 
dom snsuiudomiJ training. if» howi^va, the local ummpioynimi 

is noi a variafafe in Uie analysiH, and the amly^sd comm 
In labor shortage mm^, ihe i eturn xo on4M4ob training might fait 
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dcDrmed arras. Litewise, we raeasure relaUonships whicn ?wst 

SS^itf^ "'^y be accumie ftir the uroup undtr sludy. 
th^ way be fflcyli to iencralize, 

: benefits ai^ts m 

not measure dirwUy the effeci of cbangcs m pcogram size, wmie 
one may have considerably higher averagyatws than 

MolSS«»tag^lhe sl« of that program may lead lo smaller 
^S'sSpldoUarofcostthan-wouidp^ur^c^^ 
The Drogram with the lower average raiios. Unul further 
nl^lSs available on ma^nal gain, f 
assume that Ae effects of changing the size oi propatns vmU be . 
vdirertiy rdated to their ratios of iucres^ 

FinaUv there is a problem which <his primer hoim to solve. 
B^^^te been a prop^ ^^o^ 
^ch uslns dlffereni methodologies. ^^P'^^'^f J^^^y' 
different tyce of control or comparison group and a ajnereni way 
of haSnT^ wmUties and jKondary cfftcis. Data sources^, 
l^^?^uStrid measures of benefits will differ w^^h 
S SSent^riods wSU be used for prdjecting oenefiis and 
SuiTrates ^11 be employed for esilm.«n8 ptesem 

=r^l ^ Sels^P^io;;. and tSniques n«d .o be 
Sit ifour ho^ that if evalu^tors follow .he 
^^ures presented in thi. primer. »^cre wiU t« wffipt 
ImUarity among individufli siudJcs w make them comparable. 
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A Ftaftl Word for Itopsrt ETdttitioB 

Thf t^se of imal^ls wa have proposed is difficult to conduct 
aj^ ob^oiqsly fin^ ft may not be corrMt ^fen 

complc^. 'Wu^t MrfTMSo for its us*, however. It fdrc^ 
Sponsible for sodi propart didsions to attach weights to their 
: goals ^d to^UMdi^ the suc^s and rosts of k pro^asi as far ^ 
is possibli content with vague qua^tatij^e 

judpttenta and ptesonri hunch^. This is obviously a good thing p 
itselif; some Morattation is better than npne. Also^ it has the vejry 
valuable by-pr^uct of f*ai8ing qu would otherwise 

not have b^n asked. There is a; considerable expansion in tjie 
outlodfe ^Ith which /the programs, are viewed, from dally 
operational ^u^Uons40 the broader perspectives of social im^cts 
.and cdsts* Tftus, even though impitt evaluation may not iJiways 
Ipve the **ri|Jit" towers, it may lead to the asking of rt^re 
**fight-* qaesiionsif used sensibly* As experien?^ aijd expertise are 
awumulated, this meiho4jhould lead to better answers* / 
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fcterdje S-1 

Amy R'-hasrecantly^rapleted high school and is contemplat- 
ing her fufrdr^he hw sevwal alternatives from which to choose. 

• D She E^beVn offered a job as a sales trainee by a Iwp 
manufacturing firm. They wiU start her at a salary of $7 W 
' per year with gtiarantecd increases of Sl.OOO for the next two 
years. After tli#t point she wUl be paid commission* only^ 
Successful saleSpersohs earn $25,000 pe.r year, I 10 
percent of aU trainees will eventually fii snt.^ uas 
classification. An additional 40 percent average il 5/ M . 

yeajr. The, remainder average $10,000 per year. 

2) She can inter an apprenticeship to become, a ^"P^^"- 
Under the terms of the apprenticeship she wUl earn S3. 50, 
$4 25 and $5.00 per hour, respectively, for the three years or 
the apprenticeship and expect to worH about 1 ,800. hours per 
year When sfiJ completes tht program and receives her 

■ union card, shS will make $7.00 per hou. for 2,006 hours of 
wprk. • ' * - , ' 

3) She can enter college and study to 6e an accountant. The four 
years of school will cost $3,000 per year in tuition, room, 
board, and other oui-of-pocktt costs. . Accountams .earn, 
about $12,000 to start mS increase to average earmngs of 
$15,W0, $20,000, and $30,QC» per yeaj after five, ten, and 
iwenty years of experience, respectively. 

Ignoring her abilities and interest, what considerations should 
Amy R consider in choosing among the alternatives? Under what 
circumstances would each of the ahernatives be preferable? 



The Nation^ Institata for Conserving Energy (NICE) has 
awarded the City of Podunk a Sl,C»O,0OO grant to hire' 
utfemployed older persons to winterize and insulati homes and 
apartments in low income neighborhoods^of Podunk* The workers 
are paid the niinimum wage and the materials are purchased, 
directly from the national manufacturers at the manuficturers' 
cost.- ' ^ 

Congressman Joe Nathan who represents the district which 
includes Podunk has received tft following letter signed jointly by 
the PbdunC Insulating Contractors Association, Local 57 of the 
United Insulators Union^ and the Building Suppliers. Aisociationi 
criticidng the program: 

Dear Congressman Nathan: 

As a ch^pion of the free enterprise system, we know you 
will want tQ deal with yet another example of federal 
interference in the free market. Several months ago your office 
announced a large grant to the City of Podunk which was 
designed to reduce unemployment from its present high rnte of 
12 percent by hiring the Unemployed tp carry on insulating 
work* / > 

Mr. Congressman, this project is creating more unemploy- * 
ment thin it is solving. We insulate homes throughout the 'city, 
in all of its ndghborhoods. By the hirin| of the unemployed' 
older workers at substandard wages, the government is 
substituting their labor for those of our members. Furthermore, 
by purchasing supplies from the manufactiirers, the city is 
depriving local building suppliers from providing their normal 
services. 

The result of these actions could be ruinous to the insulating ^ 

' iridusiry of Podunk, Mr. Nathan, we are counting on you to 

stop this bureaucratic blundering and save our jobs, 

■ ^ 
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eonressmtoNBthanhasforwardedthete^^^ 

NICE.^QU have been initructfd to examine the issue and wnte a 

reply for thii IJirector. 
1) What additidnal inWnnatiop would you like to have about 
" ttie situation? - 
■2) Writeihe reply assuming that you had aU of the relevant 
focts. • / . 
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